DOCDMENT EESHME 



ED 054 236 


24 TM 000 875 


AUTHOR 


Bippey, Robert M. 


TITLE 


Scoreing and Analyzing Confidence Tests. Final 
Report. 


INSTITUTION 


Illinois Univ. , Chicago. 


SPONS AGENCY 


Office of Education (DHEW) , Washington, D. C. Bureau 
of Research. 


BUREAU NO 


BR-7-0578 


PUB DATE 


8 May 71 


GRANT 


OEG-5- 9-230578-0070 


NOTE 


87 p. 


EDRS PRICE 


EDRS Price MF-S0.65 HC-S3.29 


DESCRIPTORS 


Computer Programs, Correlation, Factor Analysis, 
♦Guessing (Tests), High School Students, Item 
Analysis, *Scores, Scoring, ^Scoring Formulas, Sex 
Differences, *Statistical Analysis, *Testing, Test 
Reliability, Tests, Test Validity, Weighted Scores 


IDENTIFIERS 


♦Confidence Tests, Item Difficulty 



ABSTRACT 



reliability and 
discussed. However, studies 
their confidence uniformly. 



Technical improvements, which may be made in the 
validity of tests through confidence scores, are 
indicate that subjects do not handle 
(MS) 




TM 00 0 8 75 , EDO 54236 



Final Report 

Project No. 7-0578 
Grant No. 5-9-230578-0070 



Scoring and Analyzing Confidence Tests 






A I 



Robert M. Rippey 

University of Illinois at Chicago Circle 
Chicago, Illinois 
May 8, 1971 



U.S. DEPARTMENT OF HEALTH, 
EDUCATION & WELFARE 
OFFICE OF EDUCATION 
THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIG- 
INATING IT. POINTS OF VIEW OR OPIN- 
IONS STATED DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY. 



i 



U.S. Department of 
Health, Education, and Welfare 

Office of Education 
Bureau of Research 



lERiC 




Final Report 



Project No. 7-0578 
Grant No. 5-9-230578-0070 



Scoring and Analyzing Confidence Tests 



Robert M. R i ppey 

University of Illinois at Chicago Circle 
Ch i cago, ! I 1 i no » s 
May 8, 1971 



The research reported herein was performed pursuant to a grant 
with the Office of Education, U.S, Department of Health, Education, and 
Welfare. Contractors undertaking such projects under Government sponsor- 
ship are encouraged to express freely their professional judgement in the 
conduct of the project. Points of view or opinions stated do not, therefor 
necessarily represent official Office of Education position or policy. 



U.S. Department of 
Heal th , Educat ion , and Wei fare 

Office of Education 
Bureau of Research 



CONTENTS 



Background 

Development of a Scoring System 
Studies of Different Scoring Systems 
Des i gn 

Results and Discussion 
I tern difficulties 
An Entropic Scoring Function 

Properties of Tests Containing Intrinsic Items 
Importance of Weighting 
The Meaning of Confidence 
Cone 1 us i ons 



Page 

1 

7 

15 

15 

16 
18 
22 
25 

25 

26 
38 



Append i x 


A. Test Scoring Programs 




Append i x 


B. Sample Items from Tests Employing 
Euclidean Confidence Scoring 




Appendix 


C. Criterion Response Distribution Plotting 
Program 


Page 


Table 1 . 


Five Scoring Functions 


4 


Table 2. 


100-point Payoff Scheme 


1 1 


Table 3. 


Comparison of Reliabilities of Scores on 





Fourteen Administrations of Three Parts of 
STEP Writing Tests, Level 1, 17 

Table 4. Mean Scores on Twelve Tests Using Four 

Different Scoring Functions 19 

Table 5. Comparison of Reliabilities of Euclidean 

Scores, eighted and Unweighted 26 

Table 6. Mean Scores on Nineteen Variables, Incentive 

Condition 28 

Table 7. Mean Scores on Nineteen Variables, Relaxed 

Cond i t i on 28 

Table 8, Mean Scores on PLN for Groups 



ERIC 









31 



Tabl e 


9. 


Correlation Coefficients for Nineteen 
Variables, Relaxed Condition 


33 


Table 


10. 


Correlation Coefficients for Nineteen 
Variables, Incentive Condition 


34 


Table 


1 1 . 


Factor Matrix for Nineteen Variables, 
Incent i ve Cond i t ion 


36 


Table 


12. 


Facotr Matrix for Nineteen Variables, 
Rel axed Cond i t i on 


37 


F i gure 


1 . 


Response Triangle for Three-Option 
Item and Scoring Key 


9 


F i gure 


2. 


Graph of Scores for Functions 1 and 2 


18 


F i gure 


3. 


Graph of Scores for Functions 1 and 5 


18 


Fi gure 


4. 


Graph of Scores Using Four Functions 


20 





4 



SCORING AND ANALYZING 



CONFIDENCE TESTS 
Robert M. Rippey 



Background 



A conventional multiple choice item is a very special case 
of a much more general situation of decision making. Given a body 
of information, knowledge of a field, a limited number of options 
to choose among, a subject will in the conventional situation: 
consider the options, develop a set of preferences, and then make 
a single choice as instructed. Such a decision, however, is not 
particularly informative about the state of knowledge of an 
individual. The single, unqualified choice does not separate the 
confident subject from the timid one. Nor does it separate the 
lucky guesser from the qualified and certain expert. Furthermore, 
the search by test makers for questions having unique correct 
responses limits them to a small fraction of the possible questbns 
which might be contrived in that broad area lying between warranted 
knowledge and aleatory opinion. Perhaps good guesses are as good 
as the same choice made with greater assurance. On the other 
hand, there may be some value in exploring more systematically 
some of the alternatives to dogmatic testing practices. 

This additional information about confidence and distribution 
of belief may be important from several standpoints. It may lead 
to more accurate prediction of retention (Ahlgren, 1968). Further- 
more, validity may be increased (Hambelton, Roberts, and Traub, 1970). 
Confidence information may be of importance in understanding the 
mechanisms involved in predicting performance in decision making 
involving complex sets of contingencies (Bruner, Goodnow, and 
Austin, 1956 ; Ward and Edwards, I 96 I; Kogan and Wallach, 1964). 

Degree of belief in the options for an item may be represented 
by a set of weights. Althouv^h a set of weights contains more 
information than a single choice, the meaning of this additional 
information must be questioned, for these weights may mean different 
things to different subjects, and most certainly will mean different 
things depending upon the conditions of administration and the 
rewards or punishment expected as part of the testing procedure. 

It is perhaps because of uncertainty about the meaning of such 
weights that test makers have preferred almost exclusively items 
which had warranted unique answers. At least examples of tests 
of another sort are difficult to find. 

By comparison with achievement tests, attitude and personality 
tests have eshewed unique right answers. The Strong Vocational 
Interest Blank, originally published in 1927> was an early example 
of an instrument Involving response wei ght i ng, (Strong , 1943). 

Others have followed (Swineford, 1941; Guttman, 1947). On the 
other hand, test makers have moved cautiously with uncertainty 




in the cognitive domain. Perhaps it is more acceptable to be 
uncertain how we feel than it is to be uncertain about what we 
know. Plato struggled briefly with the status of belief, dis- 
missing the issue as follows (Plato, Repub 1 i c ) : 

Indeed, I am content; I proceeded, to call as before the 
first division science, the second understanding, the third 
belief, and the fourth conjecture -- the two latter jointly 
constituting opinion, and the two former i rtel 1 i gence. 

Opinion deals with the changing, intelligence with the real; 
and as the real is to the changing, so is intelligence to 
opinion, so is science to belief, and understanding to 
conjecture. But the analogy between the objects of these 
mental acts, and the twofold division of the provinces of 
opinion and of intelligence, we had better omit, Glaucon, 
to prevent burdening ourselves with discussion far outnumbering 
al 1 the former. 

Although realizing that the area of uncertainty is always larger 
than the area of warranted knowledge, curriculum builders and test- 
makers have favored the certain side of the road, except in dire 
emergencies. It is interesting that much of the research in testing 
for uncertainty has come from the medical (Lewy & McGuire, 1966) 
and the military profession (Shuford, 1967)* 

In 1936, Soderquist suggested the application of confidence 
weights to true-false items and the application of penalties for 
misplaced confidence. Ebei (1965), in following this suggestion, 
obtained reliability increases of approximately .10. 

A brief inroad into uncertainty was made by the Progressive 
Education Association in its Eight Year Study (Smith & Tyler, 

19^2). Items used in their investigation of outcomes of secondary 
schooling were not only classified as correct or incorrect, but 
also “Caution," “Insufficient data," “Beyond data," “Too 
certainV and “Too uncertain." 

Dressel and Schmid (1935) contrasted the results obtained from 
a multiple choice test when administered in several novel forms. 

Two modifications were: 

A. The student was told to mark as many choices as he needed in 
order to make certain he had the right answer. This was scored 
as fol 1 ows : 



Number 


of choices 


Marked 




1 


2 


3 


4 


5 


Score 


if correct 


answer was 


marked 


4 


3 


2 


1 


0 


Score 


if correct 


answer was 


not marked 


-1 


-2 


-3 


-4 


X 



- 2 - 




B. The student was told to mark a single answer but indicate de- 
gree of certainty, using a scale of 1 to 4 meant "at least certain." 
The item was scored as follows: 



Certa i nty marked 




1 


2 


3 


4 


Omi t 


Score 


i f answer was 


correct 


4 


3 


2 


1 


0 


Score 


if answer was 


i ncor rect 


-4 


-3 


-2 


-1 





Neither of these systems actually assigned probabilities or 
degree of belief weights to the entire set of responses. 

DeFinetti (1965) has discussed a number of the consequences 
of utilizing different scoring systems and argues strongly for 
the training of subjects in a more prudent strategy for dealing 
with uncertainty. 

Feedback, or learning from experience, does not only concern 
the reinforcement of such general ideas about the profitability 
of an undistorted forecast or response corresponding to 
personal evaluation of probability: the evaluation itself 
becomes Improved by experience. It is particularly common 
for untrained people to reflect in their numerical evaluations 
of probabilities the effect of the usual way of thinking in 
rough terms of 'certain,' 'impossible,' 'unknown,' or 'com- 
pletely indifferent,' giving values 1.00 amd 0.00 to the 
favored and rejected alternatives and 0.50 in the case of 
uncertainty between two, and so on. Experience forces them 
to realize how relatively often there are events which 
occur that can be too hastily classified as impossible, and 
they learn the advantage of giving these events an adequate 
small positive probability. It Is chiefly because it provides 
the possibility of redressing such essential weaknesses in 
the machinery of human reasoning, and shows how workable 
measurement in the fields of belief can be developed, that 
I have felt obliged to emphasize so strongly the desira- 
bility of training in the use of the methods described in • 
this paper. 



Shuford and Massengill have argued that reliabilily and 
validity of tests can be increase using a class of scoring functions 
called reproducing scoring functions which maximize s score 
if and only if his responses match his internal belief state. 

These functions were first studied by Toda (1963). The data available 
at this time does not conclusively support the viability of this 
model for responses of ^s. Therefore, another purpose of this 
study was to clarify the meaning of significance of the wieghts 
which subjects assign to confidence scored tests. 
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A number of scoring functions have been developed for scoring 
confidence test items. Five such functions are shown in Table 1. 



TABLE I 

Five Scoring Functions 
Probability assigned to correct response 

S = r^ 

Logar i thmic 

If r^ .01 , S = (2 +log^Q(r^))/2 
If rj^ < .01 , S = 0 



3. Spherical 



n o 1 

^ \ 2 



S " r, / o: (r,)'- )“ 
K i.l ' 



4. Eucl idean 



S - 1 - 



n o j. 

(Z; (r, - k = 
i=1 ' 

VT 



5. Inferred Choice 



S=1 ifr, > r. forall i /k; 

k I 

S = 0 otherwise 



t h 

r. = Probability assigned to the i response 
rj^ = Probability assigned to the correct response 

th 

k. = Criterion group mean probability assigned to the i response. 



The probability assigned to the correct answer is the simplest 
and most intuitively obvious scoring function. Both the logarithmic 
and the spherical function possess the interesting property of allowing 
the student to maximize his score if and only if he does not guess 
(Shuford, Albert & MaSjengill, 1966). The Euclidean function will 
score items which do not unique correct responses. Thusitems can 
be constructed which call for answer which correspond to a distribution 



of preference representing the concensus of a group of experts. 

This function will also score items having unique correct answers. 

Inferred choice is analogous to conventional 
multiple choice scoring. According to this rule, the subject 
receives 1 point if his maximum confidence is assigned to the correct 
option. Otherwise he receives nothing. 

Although this method of scoring simulates the performance 
of subjects on conventional multiple choice tests, it does not 
duplicate it. Some subjects, confronted by absolute lack of 
preference guess. Others do not. The inferred choice function 
simulates the behavior of the subject who never guesses when he 
is absolutely uncertain, but who always makes a choice, even if 
his preference is slight. Since there are varieties of 
subject behavior on tests, this function will not always give 
results which are identical to the choices an individual subject 
might make. If a subject is instructed to answer every question, 
and if all subjects do this, scores obtained by the inferred choice 
function would always be less than or equal to scores obtained by 
subjects under the usual choice situation, since subjects would 
occasionally get an additional point due to guessing. If subjects 
were told not to guess, and the conventional penalty for guessing 
were applied, scores obtained by the inferred choice function 
would be less predictable, and could be either greater than or 
less than the scores obtained by the subject responding in the 
conventional manner. 

The seriousness of this discrepancy would be proportional to 
the number of instances on a test where no dominant preference 
was shown for a single option. In an analysis of a random 
sample of answer sheets for the STEP test data used in this 
study, such a lack of preference was found in less than 15 % ^f 
the responses. Since the item? were unusually difficult for 
the subjects by design, it is likely that the amount of guessing 
would be less on other tests, more appropriate in difficulty for 
the subjects. 

On a three option test, this would suggest that scores would 
be approximately 5% higher in terms of ^ behavior as compared with 
inferred choice scores. 

It would, of course, be possible to simulate all manner of 
erratic S behavior in responding to multiple choice items. However, 
the inferred choice function does simulate the subject who does 
not guess. Any other simulations, involving random or systematic 
awarding of points in guess situations would lead to less reliable 
scores than the inferred choice function. In comparing both the 
reliability and the validity of functions against the standard of 
choice, it is probably best to use the inferred choice function 
as a standard since it does not contain any artificially induced 
error. 



It would be possible, of course, to instruct s to record 
both their probabilities and their choices. Thus, in the event 
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of a split decision, the subject could flip his coin. Or the subject 
could also be instructed never to make his preference weights exactly 
equal. This should not be too unrealistic, since it is seldom, if 
ever, that the preference weights for a set of options would be 
entirely equal, no matter what the state of ignorance of Thus, 
although the inferred choice method does not simulate human 
behavior exactly, and this should be kept in mind, it is also 
unlikely that the inferred choice function produces less reliable 
or valid scores, o; significantly lower scores, than would be 
given by scores obtained by conventional choice methods. Since 
the purpose of much research on confidence testing is to demonstrate 
the superiority of confidence methods over the conventional choice 
method, it seems that using the inferred choice function as a 
basis for comparison does not weaken the conclusions of such 
compar i sons . 

Although many arguments and some practice accept getting a 
fix on confidence, states of knowledge intermediate between 
certainty and chaos are not as readily accepted by some subject 
matter specialists. Therefore, when one examines achievement 
tests, it is unlikely that he will find many i terns deal i ng with 
incomplete information or uncertainty. The dearth of items 
requiring a distribution of belief over the available options 
may be due to a single technical consideration. Indeed, it 
can be argued that Intrinsic items should not be written at all 
because an item which calls for a uniform distribution of confidence 
over all responses will not discriminate between the informed 
and the uninformed. Both groups would assign equal proabilities 
to all the options. Thus, the unweighted Euclidean function is 
inadequate by Itself, for items not having single option responses 
are less efficient in detecting a state of no information than 
items having unique correct options. Nevertheless, this problem 
can be rectified by asking S for a distribution of belief and 
his confidence in his distribution, and subsequently incorporating 
both the distribution and the confidence measure into his score. 

The following function accomplishes this: 



6. Weighted Euclidean function 

S = C( 1 - 2D / D ) 
max 



C = Confidence (0< C 9) 

D = Distance from s response to the criterion group response 

D = Maximum distance attainable from the criterion groupresponse 
max 



If ^s use confidence weights varying from 0 to 9» scores will 
vary from -9 to +9. An examinee who expresses no confidence will 
be neither rewarded nor penalized for his distribution of preference. 
On the other hand, certainty about a single incorrect response may 



may suffer a nine-point penalty. The results of a test containing 
a mixture of items may be scored in at least three ways: correctness, 
confidence, and appropriate use of caution. The last measure will 
be developed in a bter part of this report. 

DEVELOPMENT OF A SCORING SYSTEM 

The study of confidence scored tests requires systems for scoring 
and computing the reliability and validity of such tests. A number 
of practical matters entered into the evolution of the programs which 
are displayed in Appendix A. There are four programs in all. Punched 
output from these programs can be used in conjunction with a multiple- 
regression and factor analysis program if desired. The prograrre are 
l) a basic scoring program for eight different functions, 2) A key 
preparation program for preparation of a key from a set of responses from 
members of a criterion group, 3) A program for the weighted Euclidean 
function, and 4) A program for converting subject responses into risk 
taking indices. Descriptions of these programs follow. 

1 . The basic scoring program 

Eight scoring function are available. Option One is a simple 
Euclidean distance function. Option Two is the Spherical function. 

Option Three is the logarithmic function. Option Four computes the 
inferred choice score. Option Five applies the Euclidean function to 
items having unique correct responses. Option Six assigns a score 
to an item which is equal to the probability the subject assigns to 
the correct answer. Option Seven, the Entropic function, will be dis- 
cussed later. Option Eight is the Euclidean distance function 
with a correction for the maximum distance attainable from the 
criterion group response. 

Limi tations . 

The program will score tests of up to seventy-two items. The 
number of response options may vary from two to six. Any number of 
subjects may be scored on each pass. Input formats are fixed. 

Card Preparation. 



Card 1 Title card — Cols 2-60 

Card 2 
Col s . 

1 Standard deviation of Criterion Group Scores 

1 = No; 2 = Yes. (Applicable only to Option One) 
4 Punched output of subject scores. 

1 = Yes; otherwise bldnk. 

Card 3 
Cols . 

1 Punch a 1 in this column 

2-3 Total number of items on test (Not more than 72) 

4 Number of responses per item (2-6) 

5-6 Number of cards per subject (Not more than 10) 

7-8 Number of items per card 




r7- oil 



Function Selection Card 

Punch a one in the column corresponding to the 
functions(s) selected. 



Euclidean Function (Intrinsic Items) 

Spherical Function 
Logarithmic Function 
Inferred Choice 
Eucl idean Unique 

Probability assigned to the correct response 
Entropic Function (Intrinsic Items) 

Euclidean - Gorrectedfor Maximum Distance Attainable 
(Intrinsic Items) 

Note -- Functions are of two types, intrinsic item 
and unique correct response. Functions of both types 
may not be requested on a single pass. 

Key Card(s) For intrinsic items — use punched output from Key 

preparation Program 

For items having unique correct responses — 

Punch the number of the correct response for each 
item in the column corresponding to the number of 
the item. For example, if the correct response to 
item 7 is response option 3, a 3 would be punched 
in column ?• 

Subject response cards 

Col s . 

1-5 Identification number 

6 Card sequence number 

7 Responses to each item. The weight assigned by ^ 

to each response. 

Blank card 

Jobs may be stacked. A card with a -1 punch in columns 1-2 should 
follow the last Job. 

Output 

Output may be either printed, or printed and punched. It consists 
of individual items scores and total scores, item difficulties, and 
test reliability computed according to Hoyt's analysis of variance 
method. The punched output may be used as input to a standard stat- 
istical program for the purpose of obtaining inter-item correlations, 
correlations of items with total test and criterion scores, as well 
as factor analysis. 

2 . Key preparation from Criterion Responses 

Card 1 
Cols. 



Card 4 



Cols. 

1 

2 

3 

4 

5 

6 

7 

8 



1 



Punch a 1 in this column 



Cols. 



2-3 Number of items 

4 Number of responses per item: may vary from 2 to 6 

5-6 Number of cards per subject 

7-8 Number of items per card 

9-49 Title 

Criterion group Responses 

Blank card 

Jobs may be stacked. A card with a -1 punched in columns 2-3 should 
fol low the last job. 

Output 

Title card containing paramters for the scoring program (Card 3) i 
and a set of cards containing the mean probabilities of the criterion 
group responses. These latter cards form the key for the scoring 
program. For items having unique correct responses, the key may be 
replaced by a single card. 

3 . Weighted Euclidean Scoring Function 

This program score tests of up to thirty-two items. I terns may 
have three options followed by a fourth measure of confidence. A sample 
answer sheet and instructions follow. A key may be prepared by cal- 
culating the mean responses assigned to the options by a criterion 
group, or by means of the separate program for key preparation. Two 
scoring options are available, the unweighted and the weighted option. 
The weighted option is preferred. The unweighted option assigns a 
score to each item according to the formula; 



S = 1 - 2D / Dmax 

where d is the distance between the subjects answer (A) and the 
criterion group mean answer (B) . D max is the maximum distance 
between any point on the probabilistic triangle and the criterion group 
point. See Figure 1 . 
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In the weighted option, this score is multiplied by the degree of 
confidence, 0 through 9. 

Card Preparation. 



Parameter Card 



Col s . 
1 
2 

3 

k 

5-6 



Key 

Option One: 
Co] s . 

1-6 

7-70 



Option Two: 



Keying Option (See below) 

Punched output option 
1 = Yes; otherwise blank. 

Weighting option. 

Unweighted = 1; otherwise blank. 

1 if fewer than 17 items; 2 if 17 or more items. 
Number of items. Less than or equal to 32. 



Test number 

Key responses (Mean criterion responses). 
Format 16(I3,1X) 



Punched output from Key Preparation Program. 
Title card Cols 2-60. 



Responses 

Col s . 

1-6 

7-70 



71 



Blank card 



Identification Number 

Responses, up to 16 items, in groups of 4 
The fourth response represents confidence 
0 = No confidence, 9 “ Absolute certainty 
Test attitude 

0 = Hate the test; 9 = Love the test. 



Jobs may be stacked. 

Last job should be followed by a card with -1 punched in columns 1-2. 
A SIMPLIFIED LOGARITHMIC SCORING SYSTEM 



For persons wishing to use a logarithmic scoring system wi thout 
investing in computers or other feedback devices, the following system 
may be useful. It allows the use of the logarithmic function in a 
limited way. That is, the subject may use a fairly gross division of 



his confidence into ten categories. This approach is similar to 
DiFinetti's "Five Star" system (1965) which he used with young children. 
However, this system incorporates the logarithmic transformation, 

S = (2 + log^o^'k^ ^ ^ 

Instructions and a sample sheet for the simplified logarithmic system 
follow. If it is desired to use a more precise assignment of probabilities. 
Table 2 may be used with modified instructions. 



Table 2 



Amt . 


Amt . 


Amt . 


Amt . 


Amt . 


Amt . 


Amt . 


Amt . 


Bet 


Paid 


Bet 


Paid 


Bet 


Paid 


Bet 


Paid 


.01 


0.00 


.31 


0.7457 


.61 


0.8927 


.91 


0.9795 


.02 


0.1505 


.32 


0.7526 


.62 


0.8962 


.92 


0.9819 


.03 


0.2386 


.33 


0.7593 


.63 


0.8997 


.93 


0.9843 


.04 


0.3010 


.34 


0.7658 


.64 


0.9031 


.94 


0.9866 


.05 


0.3495 


.35 


0.7721 


.65 


0.9065 


.95 


0.9889 


.06 


0.3891 


.36 


0.7782 


.66 


0.9098 


.96 


0.9912 


.07 


0.4226 


.37 


0.7841 


.67 


0.9131 


.97 


0.9934 


.08 


0.4516 


.38 


0.7899 


.68 


0.9163 


.98 


0.9956 


.09 


0.4771 


.39 


0.7956 


.69 


0.9194 


.99 


0.9988 


.10 


0.5000 


.40 


0.8010 


.70 


0.9226 


1.00 


1 .0000 


.11 


0.5207 


.41 


0.8064 


.71 


0.9257 






.12 


0.5396 


.42 


O.8II6 


.72 


0.9287 






.13 


0.5569 


.43 


0.8168 


.73 


0.9317 






.14 


0.5731 


.44 


0.8218 


.74 


0.9346 






.15 


0.5881 


.45 


0.8266 


.75 


0.9376 






.16 


0.6021 


.46 


0.8314 


.76 


0.9404 






.17 


O.6I52 


.47 


0.8361 


.11 


0.9433 






.18 


0.6276 


.48 


0.8406 


.78 


0.9461 






.19 


0.6394 


.49 


0.8451 


.79 


0.9488 






.20 


0.6505 


.50 


0.8495 


.80 


0.9516 






.21 


0.6611 


.51 


0.8538 


.81 


0.9543 






.22 


0.6712 


.52 


0.8580 


.82 


0.9579 






.23 


0.6808 


.53 


0.8622 


.83 


0.9596 






.24 


0.6901 


.54 


0.8662 


.84 


0.9622 






.25 


0.6990 


.55 


0.8702 


.85 


0.9647 






.26 


0.7075 


.56 


0.8741 


.86 


0.9673 






.27 


0.7157 


.57 


0.8779 


.87 


0.9698 






.28 


0.7236 


.58 


0.8817 


‘.88 


0.9723 






.29 


0.7312 


.59 


0.8855 


.89 


0.9747 






.30 


0.7386 


.60 


0.8891 


.90 


0.9771 







INSTRUCTIONS 



In the following test, Imagine that you are being paid for the amount 
which you learn. However, it will cost you something to attempt each item. 
On the other hand, you may bet on more than one of the choices. 



Imagine that you have ten cents to place on the choices of each item. 

You may distribute each of these ten cents (points) in any way you like. However, 
you will only be paid on the basis of the number of points you put on the correct 
answer. The amount that you get paid can be as much as 1.00 but it might be 
as little as nothing. The amount you get paid for a given number of points is 
shown in the table below. 



Amount bet 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 



Amount paid 

0.00 

0.50 

0.65 

0.74 

0.80 

0.84 

0.88 

0.91 

0.94 

0.97 

1.00 



On your answer sheet, you will write down both how many points you bet, and also, 
what the table says you should be paid. Make certain you check to see that you 
do not use more than ten points for each item. Also, use the table accurately. 
Errors in using the table or in assigning the correct number of points will 
cause you to receive no points at all for that item. Below are some examples 
of answers to the question 



2 + 2 = ? 



1. 0 

2. 2 

3. 4 

4. 6 














Sample answers 


Student A 


student B 


Student C 


Student D 


Student E 


1 . 0 0 


0 0 


3 


.74 


10 


0 ' 


5 .84 


2. 0 0 


0 0 


2 


.65 


0 


0 


4 .80 


3. 10 1,00 


5 .84 


2 


.65 


0 


0 


3 .78 


4. 0 0 


5 .84 


3 


.74 


0 


0 


0 0 


See ff you can figure out what 


score 


each 


s tudent 


wou 1 d 


1 receive. 



Student A would receive a score of 1,00 since he knew the correct answer 
and put it down. 



Student B was unsure as to whether the answer was 3 or 4, so he bet five 
points on each and was awarded a score of 0,84 

Student C couldn't tell which answer he liked best, but was honest about it, 
and divided his points as evenly as he could across all the answers. He 
received 0,65 for his honesty. 

Student D didn't know the answer either, but he guessed at response number 
1. Since this was wrong, he got nothing at all. 



Student E also received a 0 score because he cheated. He used more than 10 
points, and also did not use the table correctly. 



On this kind of test you will make your highest score if you honestly report 
your confidence in each of the choices. The penalty for guessing is very great. 
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ANSWER SHEET 



Name 




S chool 


Date 


Question Points Score 


Question Points 


Score Question Points 


1. 


a. 


8. a. 


15, a. 




b. 


b. 


b. 




c. 


c. 


c. 




d. 


d. 


d. 


2. 


a. 


9. a. 


16, a. 




b. 


b. 


b. 




c. 


c . 


c. 




d. 


d. 


d. 


3. 


a. 


10, a. 


17, a. 




b. 


b. 


b. 




c. 


c. 


c. 




d. 


d. 


d. 


4. 


a. 


11, a. 


18, a. 




b. 


b. 


b. 




c. 


c. 


c. 




d. 


d. 


d. 


5. 


a. 


12, a. 


19, a. 




b. 


b. 


b. 




c. 


c. 


c. 




d. 


d. 


d. 


6, 


a. 


13. a. 


20, a. 




b. 


b. 


b. 




c. 


c. 


c. 




d. 


d. 


d. 


7. 


a. 


14, a, • 






b. 


b. 






c. 


c. 






d. 


d. 





Score 
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STUDIES OF DIFFERENT SCORING SYSTEMS 



Subsequent to the development of the scoring programs several 
questions were asked. These were 1) How do the several scoring ^ 

functions compare with one another and 2) Are there differences in 
the ways in which subjects respond to confidence scored tests? 

One of the fundamental assumptions underlying the logarithmic 
function, and its desirable matching property of maximizing the subject's 
score if and only if he responds with his actual degree of confidence 
is the need for the subject to have feedback at test time with response 
to the payoffs of his set of preference weights. 

Shuford has suggested several testing aids such as a computer ter- 
minal or computational devices known as scorules. Unfortunately, 
scoring in these ways in likely to be costly, or time consuming. 

(Ebel , 1968 ) Furthermore, the necessity to provide the student with 
information about the scoring system is not only demanded by theory, 
but Rippey (1968b) and Romberg and Shepler (I 968 ) both provide data 
which shows that the logarithmic function may at times produce less 
reliable scores than conventional testing when such information is 
not provided. 

An alternative to the use of computers and scorules is the use 
of scoring functions which are simpler, and more transparent to 
the intuition of the subject. The simplest of these is a score 
which is equal to the proability assigned to the "correct" answer. 

Such a score contains more information per item than a conventional 
choice score. Furthermore, it offers the student some incentive not 
to guess, although not the optimal incentive promised by the logarithmic 
function. This leads to the question of the relative merits of 
various scoring functions with respect to reliability. 

DESIGN 

A total of yjh students, hereafter referred to as Sample A, from 
three Chicago suburban high schools were given three intact 30-item 
sections of the STEP Writing Test, Level 1^, within schools. Tests 
were randomly assigned to students. No student took any form more than 
once. 



Fourteen groups were tested. Each test was scored using the five 
scoring functions. Groups 7»8| and 9 had been previously tested 
twice and had some understanding of the properties of the logarithmic 
function. Groups 4, 5i and 6 had been tested once previously. All 
other groups were totally unfamiliar with confidence tests. 

Prior to taking the test, the subjects were given the following 
statement: 



Each of the questions or incomplete statement in this test 
is followed by suggested answers. Assign a number from 0 to 9 



2 

Permission for the use of this test was granted by Educational Testing 
Service. 



to each suggested answer, depending on how strongly you feel 
the answer is correct. If you believe that only one suggested 
answer is correct, mark that answer with a 9 and mark the others 
with zeroes. If you like the suggested answers equally, assign 
the same number to each. 

Next followed several examples of how _S was to distribute the numbers 
under various patterns of degree of bel ief of the correctness of 
the several options. Finally was the statement: 

Your paper will be scored in such a way that you will get 
a higher score by estimating your degree of confidence and reporting 
it accurately. Guessing In any form will lower your score. If 
you are uninformed about the question and have no preference 
for the suggested answer, you will obtain your highest score 
by honestly distributing your confidence across all the options.... 

Tests were subsequently computer scored and relia- 

bility was estimating using Hoyt's analysis of variance procedure 
(Hoyt, 1941). This procedure is suitable for confidence tests whereas 
a number of other procedures such as K.R. 20 are not. This is because 
item scores range between 0 and 1. The Hoyt method underestimates 
reliability on short tests. Therefore the reliabilities are all 
conservat ive. 



RESULTS AND DISCUSSION 

Reliabilities obtained from the multiple scorings of the 14 tests 
are shown in Table 3. Jhe most obvious and immediate conclusion from 
these data is that the simplest of the functions, Function 1, has the 
highest reliability most of the^ time. In making the 56 possible compar- 
isons of reliabilities of scores obtained using Function 1, with 
scores obtained using another function, scores obtained by using 
Function 1 are the most reliable 49 times. Furthermore, for the 
l4 independent situations, Functioni is the best 10 times. Assuming 
a null probability of 0.2 for Function 1 to be ranked first, the 
probability of method 1 being superior 10 or more times out of 
14 i s 1 ess than . 001 . 

In order to examine the relative effectiveness of the five functions, 
an average ral lability was computed over the 14 occasions. Since 
knowledge of the logarithmic scoring function had been given to 
subjects in Groups 7, 8, and 9 on prior occasions, the second set 
of reliabilities was computed for the complementary 11 groups. The 
averages are shown in Tzble3 . 

It is clear from the table that Function 1 leads to the most 
reliable score and Function 5, conventional choice scoring leads to 
the least reliable scores. Is there some model which would explain 
this ordering of functions? Assuming that any observed score is 
made of a true component and an error component, divide the error 
in two parts. The first part is a random variable whose range is 
proportional to some discrepancy between the score assigned to a 
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Table 3 



Comparison of Reliabilities of Scores on Fourteen Administrations 
of Three Parts of STEP Writing Tests, Level 1. 



Admi n i - 

St rat ion Funct i on 



Group 


Form 


Part 


N 


1.S=Pk 


2.Loga- 3. 
r i thmic 


. Spherr 


4. Fuel i- 
dean 


5 . 1 nfer red 
Choice 


1 


1A 


1 


21 


.811^ 


.706 


.728 


.670 


.697 


2 


1A 


2 


19 


.62P 




.542 


.338 


.639 


3 


IB 


1 


22 


.6^ 


.197 


.494 


.464 


.364 


4 


1A 


1 


22 


.826 


.487 


.505 


.647 


.368 


5 


1A 


2 


19 


.811 


.060 


.469 


.740 


.691 


6 


IB 


1 


20 


.806 


.594 


.700 


.672 


.685 


7 


1A 


1 


19 


.627 


■m. 


.644 


.581 


.555 


8 


1A 


2 


23 


.m 


.489 


.366 


.539 


.305 


9 


IB 


1 


20 


.693 


.563 


.493 


.483 


.524 


10 


1A 


1 


38 


.169 


.439 


.039 


.184 


-.063 


11 


1A 


1 


39 


.641 


.525 


.489 


.595 


.521 


12 


1A 


1 


41 




.588 


.621 


.699 


.604 


13 


1A 


1 


52 


.687 


.533 


.561 


.§21 


.535 


14 


1A 


1 


23 


•Zi2 


.000 


.160 


.613 


.202 


Average^ 


four 


14 Groups 




.69 


.50 


.49 


.58 


.47 


„ b 

Average 


for 
















Groups 1 


-6, 


10-14 




.68 


.47 


.49 


.60 


.48 



0 

The largest reliability for each occasion is underscored. 

The customary £ to z transformation was used in computing the average cor- 
rel at ions . 




subject for an item, and the subject's perception of the score he will 
receive for an, item. The second part of the error then consists of the 
balance of uncontrolled variation which contributes to the unreliability 
on conventionally scored tests. Thus, most scores computed using eso- 
teric scoring functions will have an error component which is due to 
the subject's lack of understanding of the scoring system. This error 
component is illustrated in Figures 1 and 2. If a subject is told or 
believes that the score he will receive is equal to the probability he 
assigns to that option which happens to be correct (this is the intuitively 
obvious conclusion), and his ^cpire is then computed by some other function, 
then a portion of the er rorffio^pbnent of his score will be proportional 
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to this discrepancy. The discrepancy is shown in the two figures as the 
shaded area between the two curves. 

This suggests that, other sources of error being equal, (the same 
subjects take the same tests on the same occasion), the rank order of 
reliabilities will be the same as the rank order of the discrepancies 
between Function 1 and the other functions. Such, in fact was the case. 





Figure 2 Graph of Scores for Functions 
1 and 5. 



The discrepancy between scores obtained from Function 1 and scores 
obtained from Functions 2, 3, 4, and 5 were obtained by integration of 
the various functions. The results of this integration were .06, .23, 

.25 and .28 for Functions 2 through 5 respectively. (The discrepancy is 
0 for Function 1 as this function served as the standard.) Not only are 
the discrepancies and the reliabilities in exact reverse order, but also, 
the relationship between the discrepancies and the averaged reliabilities 
is essentially linear. 




ITEM DIFFICULTIES 

The simplest function, Function 1, produces a consistently high 
reliability. The Euclidean Function, which is similar to Function 1 
in its scoring results, produces a compar i t ivel y high reliability. 

However, the probability assigned to the correct answer has a 
serious disadvantage. Williams and Millman (1970) pointed out correctly 
that the simple probability function penalizes the student who honestly 
reflects his degree of belief and favors the student who shifts his 
confidence entirely to the option he prefers, even it his preference is 
slight. Therefore, with practice, a subject will learn to distort his 
responses in the direction of dogmatic choice. The objective of this 
portion of this study was to see if the Euclidean function might perhaps 
resolve this problem of confidence testing. 

-, 8 . 22 
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Two kinds of scores may be given ^s. The first is a score of correct- 
ness based on the distance of s response from the response pattern 
agreed upon as correct. The second score, a score of appropriateness, 
is a count of the number of items to which ^ responded with certainty 
when certainty was called for and caution where it may be assumed his 
knowledge was incomplete. 

Method 

A shortened version of the STEP Writing Test Level 1A was administered 
to twelve groups of high school students (N = 303). ^s were told to 
honestly represent their confidence on their answer sheets. Incentives 
were not mentioned. The results were scored using four different 
scoring functions, the simple probability, Logarithmic, Euclidean, 
and Inferred Choice functions. 

Concl us ions 

As predicted, the Logarithmic function gace the highest mean scores 
for all twelve groups and the probability assigned to the correct answer 
gave the lowest scores consistently to all twelve groups. On the other 
hand, the mean group scores obtained using the Euclidean function were 
slightly higher in four instances and slightly lower in eight instances, 
than the score derived by assigning full confidence to the preferred 
option (Inferred Choice score). These scores are displayed in Table 4. 



Table 4 

Mean Scores on Twelve Tests Using Four Different 
Scoring Functions 



Rank 

Order 


N_ 


Probab i 1 i ty 
Ass i gned 
To Correct 
Answer 


Log 

Function 


1 nf erred 
Choice 


Euci idean 


1 


22 


9.77 


18.91 


12.23 


12.54 


2 


19 


10.37 


18.98 


14.46 


12.92 


3 


20 


10.82 


18.99 


13.60 


13.45 


4 


20 


1 1 .07 


19.28 


13.70 


13.11 


5 


21 


11.09 


18.83 


15.75 


13.96 


6 


19 


11.39 


19.55 


13.31 


13.73 


7 


23 


11.79 


20.02 


14.82 


14.17 


8 


19 


11.95 


19.76 


15.53 


14.01 


9 


22 


12.00 


19.54 


14.85 


14.11 


10 


41 


13.32 


19.44 


14.25 


14.89 


11 


39 


15.72 


^ .-t 

21 . 1 3 


16.55 


16.68 


12 


38 


19.95 


24.57 


20.99 


19.95 



O 
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MEAN SCORE FOR EACH GROUP 
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Graph Of Scores Using Four Scoring Functions 
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1 2 3 4 5 6 7 8 9 10 11 12 



RANK ORDER OF GROUP MEAN SCORES USING PROBABIi.lTY 
ASSIGNS TO CORRECT ANSWER AS THE SCORING FUNCTION 



Thus the Euclidean function did not consistently penalize the 
honest student, although it did not reward him, as did the Logarthmic 
function. The high reliability of the Euclidean function therefore 
suggests that it might be a preferred method of scoring confidence 
weighted tests. It has the further advantage of being applicable to 
items not having unique correct responses. Therefore, it can be used 
with a wider variety of test items than any of the other functions. 
Furthermore, a test consisting of a mixture of items having unique 
correct responses saves the test constructors from the ethical problem 
of asking the subject to be honest about his belief when following such 
instructions may lower ^s score. 

It is true that all of the scoring functions witl rank the subjects in 
approximately, although not identically the same order. However, if a 
subject ignores the instructions of the proctor and refuses to follow 
the instructions to assign probabilities and instead gives his full 
weight of choice to his preferred option, even though that option is 
only slightly preferred, he will get a higher score. The point of the 
Euclidean and the Logarithmic functions is that they do not favor the 
subject who ignores the instructions to assign his degree of belief to 
his options. 

If, thus far, it seems that I am recommending the Euclidean function 
on all occasions, let me say that I am not. The situation as I see it 
i s as fol 1 ows . 

The Logarithmic function is recommended on the basis of its repro- 
ducing property of maximizing scores if and only if ^ assigns his degree 
of belief to the options accurately. The theory is based upon the 
maximization of ^'s expected utility for his response spectrum. This 
assumes, however, that the utility of a response is proportional to the 
score ^ receives. If in the case where S is being paid dollars for the 
points he earns — fine. However, there are many testing situations where 
utility is not proportional to the test points. In research, for example, 
if tests are not counted toward student grades, or if they receive no 
extrinsic or intrinsic rewards for their performance, utility of response 
is independent of score. In such situations, it is likely that the use 
of the probability assigned to the correct answer as a scoring function 
might be preferred because of its high reliability with naive ^s. 

Nevertheless, where the stakes are high, where ^'s career or income, 
or the fate of his regiment depend upon an accurate assessment of his 
state of knowledge, the logarithmic function is to be preferred in spite 
of the cost and trouble it entails. 

On the other hand, the Euclidean function does not deter ^ from 
reporting his probabilities as does the funct ion wh ich scores proportionally 
to the probability assigned to the correct answer. Its high reliability 
with naive _S s , and the other advantages mentioned do not make it a panacea 
to the problems of confidence testing, but the advantages do make it a 
reasonable alternative in some situations. 
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Not all knowledge is warranted knowledge. Training and education 
must prepare subjects for situations involving partial knowledge and 
incomplete information. However, technical problems have thus far 
impeded the development of tests which reflect such situations. The 
Euclidean function can achieve high reliability on conf i dence we ighted 
items without expensive or complex feedback devices, without penalizing 
the honest students, and without compromising the integrity of the test 
constructor. 

An Entropic Scoring Function 

The Euclidean function seems to lead to high reliabilities, does 
not penalize ^s for respondi ng wi th their degrees of belief. It has 
an intuitive appeal. Nevertheless, it is not tied in with the main- 
stream of information theory, nor does it provide any particular incen- 
tive for ^ to struggl e wi th his probab i 1 i t ies. 

Although the feedback devices required to bring about an accurate 
application of the Logarithmic function may be time consuming and moderately 
expensive, perhaps Ebel's point is unduly negative. For the average class- 
room teacher, perhaps accurate assessment of subjective probabilities is 
unimportant. On the other hand, to understand subjective probabilities 
and to interpret their meanings in terms of human behavior, it may be 
premature to rule out the development of their assessment until we know 
much more about them. Therefore, it may be desirable to proceed with 
the extension of such assessment without regard to simplicity. 

There are other ways of matching probabilities than comparisons of 
Euclidean Distances. For example, th6 product of the vector of subject 
responses and criterion responses, T • Tcj ranges between 1 and 0 depending 
on the closeness of the two vectors. On the other hand, the product 

,1 n 

k • (log r) = kj logr- 

n 

becomes ^ ! r. logr. when k, = r,. (1) 

'III ' 

. (1) is the formula for the entropy of a logical spectrum (Watanabe, 
1969 a). Gibb's theorm states that, given two sets of real numbers, -[k.j- , 

{r .■}■ , where i = 1 ,n and k, > 0, r j i 0, and ' 




and equality holds only where kj = r.. 
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This formula may be applied to a multiple choice Item as follows; 
Given a set of criterion response probabilities to a logical spectrum A, 



a logical spectrum can be established by providing the option "some other 
answer" in an intrinsic type item, or by informing the subject that there 
is one and only one correct answer for a conventional item. This latter 
Information makes it possible for the subject to conclude that not only 
is there One and only one correct answer which is given to him as an option, 
but also that none of the other options, or for that matter any other 
option not stated is open to him as a choice. 

If the subject's score is then made equal to 



it immediately follows from Gibb's theorem (Watanabe, 19^9 ) that this 
score is maximized only under the condition that the subject's probability 
coincides with the criterion group probabilities^. Furthermore, if an 
item has a unique correct response, the vector k. will contain a single 
1 and all other entries will be 0. In this case, then, the scoring func- 
tion becomes equivalent to the logarithmic scoring function. Thus the 



having unique correct responses. 

However, the question still remains as how to give ^ enough informa- 
tion about the system that an accurate assessment of his subjective 
probabilities may be made. It must be remembered that Toda's original 
work ( 1963 ) postulated a computer giving the ^ information about the pay- 
off of his responses, and that this payoff, depending on the scoring func- 
tion, might not be proportional to his response probability. The ^ would 
then be allowed to modify his probabilities to achieve a maximum expected 
utility (Ramsey, 1931). 

In order to receive appropriate feedback, the _S taking an h option 
item is given n payoff values, one for each of the vertices of the response 
n-hedron. In the case of an intrinsic item, the subject could be given 
the same kind of information for any number of points on or within the 
response n-hedron, including the vertices. On the other hand, the sampling 
could be left under the control of the subject who could request a computa- 
tion of his payoff based on. his own response and a variety of assumed 
criterion responses which he might choose to experiment with. 



•{ k.^ , and a set of subject responses to the same logical spectrum, {r.}. 



n 



n 



where 




n 




k. log r. 



I 



entropic scoring function is appl icabl e both to intrinsic.i terns and items 
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This may seem like a complex process. On the other hand, it is 
probably realistic since accurate probability assessment in the face of 
incomplete information is in most cases difficult, and in many cases not 
even rational. On the other hand, it seems that the number of points 
to be tested for feedback from a three-option intrinsic item would not 
be unreasonably greater than the number of points to be tested on a six- 
point conventional item. On the six option conventional item, each item 
would need feedback concerning six vertices. On a three option intrinsic 
item, one could probably get by nicely with six points — the three vertices, 
the centroid, and two other interior points. 

One characteristic of the entropic scoring function is that ^ would 
get a very low score (minus infinity) for neglecting a vertex which had 
any possibility of being correct. For example, if the criterion response 
vector were (1, 0, 0) and one chose as his probabilities 



Although this may seem like an extreme penalty, it is perhaps realistic, 
and certainly is in keeping with the methods of science. Since, as Platt 
( 1963 ) suggests, "Any conclusion that is not an exclusion is insecure and 
must be rednecked." Shuford, Albert, and Massengill (1966) were sufficiently 
uncomfortable with such a severe penalty that they limited the minimum score 
that coutd b6 obtained with an item. This is partly a matter of taste, 
and partly a question of one's objective in testing. It is not uncommon 
in. medical school examinations to have an item or so with a response which 
results i n «.f all ure. of. the entire test. Perhaps if the captain of the 
Titanic, at^, the .pommanding officer at Pearl Harbor had been tested in this 
way while in'' col lege, they would have been less likely to completely rule 
out the possibility of icebergs that far South or Japanese planes that far 
East. 

All entropic function scores are negative or zero. If this is bother- 
some, a positive constant could be added, and a lower limit could keep 
the scores within a range of +1. In the computer program, the following 
scoring function was used as Option 7 . 



0 



2 



2 



h i s score would be 



Log 0 



k * log r = (1 , Q, 0) Log i 

Log k 



- <P 



n 



S = 10. + 




logr . 



i=1 

with a minimum of 0 applied to 




Properties of Tests Containing Intrinsic I terns 



The data reported earlier utilized questions which had unique 
correct responses. Items having such answers have been used for a long 
time. Intrinsic items which do not have unique correct answers have not 
been used to any appreciable extent. In fact, such items are not easy 
to find. Several tests of this sort were produced in the course of this 
investigation. One written by the author, one written by Sallie Churchill, 
and one written by Leona Peterson may be found in Appendix . It is not 
always easy to find a sample of subjects for such items. Therefore, in 
some of the data, intrinsic items were simulated using very difficult items 
from the STEP Writing Test, Level 1, by instructing ^s that the items might 
or might not have unique correct responses. In the data reported, samples 
will be designated by numbers. Sample 1 was a group of 125 freshmen from 
one of the Chicago Junior Colleges. These subjects took the 15 simulated 
intrinsic items. Samples 3 and 5 also took the 15 simulated intrinsic 
items, ^s were sophmores and juniors from a suburban high school. Sample 
2 consisted of 263 students in Graduate Social Work programs in five 
universities. These ^s took the Test of Troublesome Management Situations 
which is found in the appendix. This test has two separate keys, with 
one set of responses representing a diagnostic-treatment approach to social 
work, and another set of responses representing a mediating approach to 
social work. The third option represents a nondescript response, not based 
on any particular philosophical or theoretical position. 

In all four samples, ^s were asked to respond to each item with a 
distribution of preference. In addition, they were asked to make an inde- 
pendent assessment of their conf icfence in this distribution. The items 
were scored using both the weighted and the unweighted Euclidean functions. 

Importance of Weighting 

Whether or not confidence weighting is useful is a question which is 
not yet clear. Tests containing intrinsic items should probably be scored 
with confidence weighting until evidence shows that the confidence weighting 
is unnecessary. This study developes some empirical evidence that weighting 
may not be necessary. 

In an analysis of the responses from several of the data samples, 
weighting had practically no effect on reliability. This lack of difference 
is shown in"'Table 5 . 

Thus, although the reliability is consistently higher with weighting, 
the difference is very slight. 

The reliability of the writing tests may seem low. Howevqr, the 
Spearman Brown correction for test length increases the reliability of the 
least reliable set of scores (.347) to 0.68, which is not far from the 
reported reliability of 0.74 for STEP Level 1 (Educational Testing Service, 
1957 ). The maximum corrected reliability for the sample 4, weighted, is 
0.80. 
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TABLE 5 





COMPARISON 


OF RELIABILITIES OF 


EUCLI DEAN 


1 SCORES, 








WEIGHTED AND UNWEIGHTED 


Rel lability 


Reliability 




Sample 


Test 


N 


Weighted 


Unwe i qhted 


1 


Jun ior Coll ege 
Freshmen 


1 5 1 tern Writing 
Test 


125 


.396 


.347 


2 


Graduate Social 
Work Students 

High School 
Sophmores and 
Juniors 


24 Item Problem 
Management Test 


297 


.755 


.743 


3 


Sample A 


1 5 1 tern Writing 
Test 


263 


.401 


.396 


4 


Sample B 


1 5 1 tern Wr i t i ng 
Test 


135 


.493 


.478 



The Meaning of Confidence 

In order to explore the nature of confidence ratings which assign 
to test items, a variety of data were used by Grozelier for her master's 
paper (1970)- Two hundred sixty-three Sophomore and Junior students from 
a high school in a suburb of Chicago were randomly assigned to two groups 
and administered the achievement test in writing described previously. 

One group was told that the test they were taking would count toward their 
grade in English. The other group was told that the test was being admin- 
istered for research purposes and would not be counted on their grades. 

The teachers were given the grades of the subjects in the incentive group, 
and they tiad agreed to utilize them in grading, although the amount of 
weight to be given to the results was not specified. They were instructed 
in the system of scoring to be used as follows; 

Each of the questions in this test is followed by suggested 
answers. Assign a number from 0 to 9 to each suggested 
answer, depending on how strongly you feel that the answer is 
correct. If you believe that only one suggested answer is 
correct, mark that answer with a 9 and mark the other (s) with 
zeros. If you like the suggested answers equally, assign the 
same number to each. The sum of the three responses should 
add up to 9. . . 

Then additional information was given by the test supervisors on the way 
it was going to be scored: 

If your answer is closer to the right answer, you will get a 
positive score. If it is closer to the wrong answer you will 
get a negative score. The scores vary from -1 to +1. They 
are multiplied by your certainty (answer D) . 
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The test itself was preceded by a six item practice test at the end of 
which subjects were given the right answer for each question and could 
ask any question about the instructions. They were told that for the 
practice test there was one single right answer, but for the test itself, 
there might or might not be more than one single right answer to each 
item. 

Grozelier's data and additional variables were used in the analysis which 
follows. In addition to the achievement test, ^s were asked to fill out a ,, 
personal data sheet, and were given a test of 5 personality variables.' 
From these instruments the following variables were measured: 

1. Sex: Male = 1, Female = 2 

2. Year in School: 1 = Sophmore, 2 = Junior 

3. Score: Mean weighted Euclidean score on the 15 item writing test 

4. Attitude: 0 = maximum dislike for test, 9 = maximum liking 

5. Confidence: 0 = minimum confidence in responses, 9 = maximum 

6. Autonomy: Scale score from Personality Research Inventory 

7. Harm Avoidance: Personality Research Inventory 

8. Impulsivity: Scale from Peraonality Research Inventory 

9. Order: Scale from Personality Research Inventory 

10. Succorance: Scale from Personality Inventory 

11. Social Class: (on a three-point scale) Low ■ 1, Middle = 2, Upper 

12. Appropriateness of Confidence (WPLN) 

13. Propensity to gamble (PLN) 

14. Appropriateness of Confidence on an item of medium difficulty 

15 . Gambling propensity on an item of medium difficulty 

16. Appropriateness of Confidence on an easy item, #7 

17 . Gambling propensity on an easy item 

18. Appropriateness on a difficult item, #13 

19 . Gambling propensity on a difficult item. 

Some explanation is necessary on the computation of variables 12 
through I 9 . 



The propensity to gamble, PLN, for an item was equal to the sum of 
the squares of the differences between numerical response for each of the 
responses and three, divided by six. That is, 



where 0 < 
and 



j=l 



r . 

J 



r , 

J 



^ 9 
=9, j 



PLN 



( (r , 

J=1 J 



= option number 



“3)^)/6 for the i'"^ item. 



Since subject responses ranged from 0 to 9 for the three options, ^ 
who had no preference for the options, and who expressed this lack of 
preference by responding (3,3,3) to the three options would receive a PLN 



☆Scales Au, Ha, Im, Or, Su, from Douglas Jackson, Personality Research 
Form, Form AA, Research Psychologists Press Inc. 1965. 




rs 

u 



-27-S; 



/ r, 








TABLE 
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INCENTIVE GROUP 










VARIABLE 


MEAN 


ERROR 


N 


ST. DEV. 


ERROR 


.1 


Sex 


1.545 


0.048 


110 


0.500 


0. 


2 


Year 


1.664 


0.045 


110 


0.475 


0.015 


3 


Score 


4.344 


0.306 


110 


3.209 


0.197 


4 


Att i tude 


3.336 


0.230 


110 


2.409 


0.123 


5 


Conf i dence 


7.912 


0.085 


110 


0.891 


0.089 


6 


Autonomy 


9.229 


0.345 


109 


3.597 


0.193 


7 


Harm Avoidance 


7.321 


0.299 


109 


3.118 


0.179 


8 


Impul s i vi ty 


10.376 


0.291 


109 


3.036 


0.181 


9 


Order 


10.000 


0.370 


109 


3.866 


0.237 


10 


Succorance 


9.899 


0.381 


109 


3.979 


0.198 


1 1 


Social Class 


1.764 


0.062 


110 


0.649 


0.047 


12 


Approp. Confidence 


1.527 


0.071 


110 


0.739 


0.053 


13 


Propensity to Gamble 


7.130 


- 0.092 


110 


0.963 


0.069 


14 


Approp. on. Med. Diff. 


2.447 


0.222 


110 


2.328 


0.119 


15 


Gamble on Med. Diff. 


6.571 


0,319 


110 


3.349 


0.117 


16 


Approp. on Easy Item 


0.292 


0.090 


110 


0.942 


0.232 


17 


Gamble on Easy Item 


8.890 


0.070 


110 


0.730 


0.283 


18 


Approp. on Hard Item 


2.174 


0.218 


110 


2.291 


0.164 


19 


Gamble on Hard Item 


6.346 


0.298 


110 


3.126 


0.150 
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RELAXED 


GROUP 










VAR TABLE 


MEAN 


ERROR 


N 


ST. DEV. 


ERROR 


1 


Sex 


1.477 


0.048 


1 1 1 


0.502 


0. 


2 


Year 


1 .631 


0.046 


1 1 1 


0,485 


0.012 


3 


Score 


3.680 


0.344 


111 


3.619 


0.229 


4 


Att i tude 


3.027 


0.244 


1 1 1 


2.574 


0.122 


5 


Conf i dence 


7.324 


0.149 


111 


1.570 


0.232 


6 


Autonomy 


9.410 


0.355 


105 


3.642 


0.185 


7 


Harm Avoidance 


7.571 


0.355 


105 


3.642 


0.198 


8 


Impul s i vi ty 


1 1 .448 


0.298 


105 


3.051 


0.186 


9 


Order 


8.667 


0.415 


105 


4.251 


0.263 


10 


Succorance 


9.371 


0.349 


105 


3.574 


0.253 


1 1 


Social Class 


1 .982 


0.067 


110 


0.704 


0.063 


12 


Approp. Confidence 


1.638 


0.119 


1 1 1 


1 ,256 


0.175 


13 


Propensity to Gamble 


6.586 


0.116 


11 1 


1 .218 


0.095 


14 


Approp. on Med. Diff. 


2.432 


0.239 


1 1 1 


2.521 


0. 176 


15 


Gamble on Med. Diff. 


5.414 


0.328 


11 1 


3.454 


0.120 


16 


Approp. on Easy Item 


0.383 


0.142 


1 1 1 


1 .501 


0.354 


17 


Gamble on Easy Item 


8.482 


0.186 


1 1 1 


1 .960 


0.358 


18 


Approp. on Hard Item 


2.220 


0.218 


1 1 1 


2.296 


0.216 


19 


Gamble on Hard Item 


5.241 


0.329 


1 1 1 


3.469 


0.115 
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equal to zero. On the other hand, ^ showing a complete preference for 
a single option (propensity to gamble) would receive PIN = (36 + 9 + 9)/6 = 9. 
Thus PIN is an index of the subject's tendency to select a single option. 

PIN for a test would then consist of the average value of PIN over all 
the items. 

Appropriateness of conf i dence^compares ^'s PIN with his expressed 
confidence in the item. For the i — item, appropriateness of confidence 
(WPLN) is the absolute value of the difference between s PIN for that 
item and his confidence measure, D. : 



Theoretically, a person with no knowledge should declare D. = 0 and 
distribute his responses (3,3,3). This would make PIN = o'and D = 0. 

Thus a score of 0 on WPLN indicates congruence between PLN and D.. 

An ^ who is certain of his response would mark one option with a 
nine and the other options with zeroes. This would make PLN = 9. 

If he was that certain, he should also mark D = 9, again giving WPLN = 0. 
Larger values of WPLN indicate a discrepancy between confidence and 
one's behavior in distributing his responses. 

Means and standard deviations of the 19 variables under the 
relaxed and the incentive conditions are shown in Tables 6 and 7- 

The reliability of the test under the i ncent i ve cond i t ion was 0.261. 
Under the relaxed condition it was 0.493. Although the mean scores 
were significantly higher under the incentive condition, the reliability 
of these scores was consistently lower. 

_Ss reported a slightly more favorable attitude toward the test 
under the incentive condition, although the average liking in both 
groups was indicative of mild displeasure. In the incentive group, 
there was a significantly greater amount of confidence than there was 
in the relaxed group, along with a significantly higher propensity to 
gamble. Thi§ was perhaps due to the greater desire of these ^s to 
improve their scores, for the confidence expressed in the incentive 
group was more congruent with their distribution of preference than 
was the confidence expressed by the relaxed group. Confidence was 
most appropriate on the easy item, and was least appropriate on the 
item of moderate difficulty. 

In her master's paper, Grozelier (1970) made a further breakdown 
by sex and social class. The following interpretation of these data 
is taken directly from her paper. 

Sex. Examination of the data indicates that girls were slightly 
more sensitive to the incentive effect than were boys (the 
increase in the girls PLN mean is significant at .02). With regard 
to the level of risk taking thereare no important differences 
coming out but at the item 13 where boys appear rather conservative 
and girls high risk mhded. It follows from our basic assumption 
that the motive to aphieve success would be stronger' among 
boys whereas girls would rather be failure avoidance oriented. 



WPLN 



PLN. - D 




Obviously this assertion is not absolutely reliable; an inde- 
pendent and simultaneous measure of the achievement motive would 
have been sounder. This observation might be accounted for by 
cultural factors, in their education boys are subject to stronger 
emphasis to reinforce success achievement trends than are girls. 

On item 6, higher class subjects appear to oe more conservative 
than the other two class students. This is particularly conspic- 
uous under the incentive condition (PLN. mean = 5.1 for the 
higher class, PLN. mean = 5-9 for the middle class, PLN. mean 
= 6.1 for the lov'sr class). Unfortunately some looseness due to 
uncertainness in the sample does not allow to estimate the difference 
between middle class and lower class as significant. But if we 
assume this difference as significant we would be confronted with 
a trichotomous situation, middle class appearing success achieve- 
ment oriented and the two other classes as fear of failure 
motivated. It should also be noted that at this item which rep- 
resents the average level of difficulty of the two other classes, 
were not sensible to incentive effect whereas the lower class 
felt it strongly (PLN. increase of 1.0). Suprisingly the neutral 
condition determined a slight increase in the middle class level 
of risk taking. 

In short, middle class subjects appear as moderate risk takers 
and therefore qualify as motivated to achieve success whereas 
lower class. . .fear of failure oriented. These assertions are 
supported by the results but the upper class pattern is not so 
clearly defined. On the whole test and on the particular item 6 
upper class appeared conservative in risk taking but in the 
neutral condition and on item 13 they qualified as fear of failure 
or i ented . 

It follows that middle class students fared the best in the con- 
fidence testing situation, their PLN indexes stand at the inter- 
mediate level and they get sco“res slightly higher than the two 
other classes (though it was not statistically significant). On 
the other hand they tend to display a motivation to achieve success. . 
It seems that the cultural and educational training which charac- 
terizes middle class youngsters would give a pr i v i 1 i gi ated prepara- 
tion to face this confidence testing situation. Strodtbeck (1958) 
in his investigation of childhood training and achievement described 
the emphasis put upon early mastery by children. Youngsters are 
trained -to be able to adapt themselves quickly to a new situation 
and be able to get the best of it; they are ready to work longer 
to succeed in a task and, as they are rather ego-oriented, personal 
realization represents more value to them than a good grade. 

Lower class students fared the worst on this test. They were more 
risky minded and tended to get the lowest scores. This finding 
can be interpreted as an issue of confidence testing procedure 
which emphasizes the importance of using partial knowledge and 
uncertainty; guessing is penalized but acknowledgement of partial 
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TABLE 8 



GROUP MEANS 



TOTAL SEX GRADE SOCIAL CLASSES 



1 

1 


1 ^ 


F ! so 


ju ! 1 


II III 





1 

1 NEUTRAL 
1 CONDITION 


6.5 


1 

1 

1 6.4 


6.7 


1 

1 

1 6.4 


6.6 1 


6.2 


6.6 


6.9 


PLN 

MEAN 


1 

1 INCENTIVE 
1 CONDITION 


7.0 


1 

1 

1 6.9 


7.2 


1 

1 7.1 


7.0 1 


7.1 


7.0 


7.8 




1 

1 

1 TOTAL 


6.8 


1 

1 

1 6.6 


6.9 


1 

1 6.7 


6.8 1 


6.8 


6.8 


7.3 




1 

1 NEUTRAL 
1 CONDITION 


8.5 


1 

1 

1 8.6 


8.4 


1 

1 8.7 


8.4 1 


8.7 


8.6 


8.2 


PLN 

ITEM 7 


1 

1 INCENTIVE 
1 CONDITION 


8.9 


1 

1 

1 8.9 


8.9 


1 

1 8.8 


8.9 1 


8.9 


9.0 


8.3 


(Easy) 


1 

1 

1 TOTAL 


8.7 


1 

1 

1 8.7 


8.7 


1 

1 8.7 


8.7 1 


8.8 


8.7 


8.3 




1 

1 NEUTRAL 
CONDITION 


5.5 


1 

1 

1 5.6 


5.3 


1 

1 

1 6.3 


5.1 ! 


5.2 


5.5 


5.1 


PLN 

ITEM 6 


1 

1 INCENTIVE 
1 CONDITION 


5.8 


1 

1 

1 5.0 


6.3 


1 

1 

1 5.1 


6.1 1 


5.1 


5.9 


6.1 


(Average) 


1 

1 

1 TOTAL 


5.6 


1 

1 

1 5.4 


5.9 


1 

1 

1 5.7 


5.6 1 


5.1 


5.7 


5.6 




1 

1 NEUTRAL 
1 CONDITION 


5.4 


1 

1 

1 4.5 


6.3 


1 

1 5.8 


5.0 1 


4.6 


5.5 


5.4 


PLN 

ITEM 13 
(Di ff icul t) 


1 

1 INCENTIVE 
1 CONDITION 


6.1 


1 

1 

1 5.4 


6.6 


1 

1 

1 6.4 


6.0 1 


6.9 


6.0 


5.9 


1 

1 

1 TOTAL 


5.7 


1 

1 

1 4.9 


6.4 


1 

1 

1 6.1 


5.5 1 


6.1 


5.7 


5.7 
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knowledge is rewarded. This test emphasizes adjustment and 
compromise but lower class people take higher risk because they 
precisely do not have the ability to compromise. This hypothesis 
has already been suggested by Bourdieu (1964) who pointed out 
this particular incapacity, affecting lower class people, to 
stand on an intermediate position. He showed how lower class 
children in the class room have only two extermes of attitude at 
their disposal, to recognize coarsely either their ignorance 
or their knowledge without any continuum in their attitude as 
the student lives in a two pole world limited to right and wrong. 
These students never learned how to scale a lack of knowledge 
by a skillful utilization of their available information because 
this would require a higher command of language which they do 
not learn at school. Hence one may conclude that many lower class 
students are not equipped to cope with this new kind of testing. 

They do not have the necessary skills. 

Higher and lower class students could be labled as fear of failure 
motivated (taking extreme risks). . .the fear of failure motiva- 
tion acts as an obstacle which keeps them from fitting into the 
new system. But the reason why this motive is working this way 
probably brings us back to some sociological factors. Nevertheless 
higher class people might get the advantage from their family milieu, 
being used to a more intellectual environment could help them to 
understand the complex presentation of the test. Of course this 
works just the opposite for lower class people who appeared 
particularly disadvantaged at this test; they tend to get the 
lowest ' scores on the whole test but the discrepancy becomes really 
striking when considering the neutral situation where they get 
a score mean of 1.9 against a score mean of 3.2 among the higher 
class and a score mean of 3.6 among the middle class. 

Correlations were computed for each of the two samples for all 19 
variables. The correlation matrices are shown in Tables 9 and 10. 
Correlations larger than r = .195 will be examined. For a single pair 
of variables, a correlation of 0.195 indicates a significant departure 
from 0.0 at the 0,025 level with 100 degrees of freedom. (Walker and Lev, 
1953). Comparing significant correlations in the two matrices, it can be 
seen that there was a significant relationship between sex and attitude 
toward the test with the girls liking it better than the boys. This sex 
difference was accentuated under the incentive condition. The males 
were more Autonomous and less Succorant in both groups. This should be 
expected because the personality test was not involved in the incentive 
instructions. Finally, only the difficult item provided a significant 
correlation with appropriateness of judgment of confidence and the pro- 
pensity to gamble with the females showing a greater willingness to make 
extreme choices, and also exhibiting greater congruence between their 
feelings of certainty and their behavior in responding to the items. That 
is, the females were more inclined to choose single responses, but they 
also felt more certain about their choices than did the males. Confidence 
was significantly related to score under both conditions, though the 
relationship was higher under the relaxed condition. That is, subjects 
were more willing to take extreme positions under the relaxed condition. 



TABLE 9 

CORRELATION COEFFICIENTS - RELAXED CONDITION 
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Thus the subjects seemed. to be more motivated by fear of failure than by 
potential reward. It is of additional interest to note that there was 
no relationship between score and the gamble score on the easy items in 
the incentive condition, while the significant relationship was on the 
hard item under the incentive condition. In fact, the gamble hard score - 
score correlation changed sign going from the relaxed condition to the 
incentive condition. That is, for the high scoring ^s , there was a 
tendancy to assume extreme positions on the hard items under the relaxed 
condition, but an unwillingness to do so under the incentive condition. 

That is, where grades were at stake, the high scoring ^s played the 
caut'ious role. s attltu^de toward the test was related primarily to his 
confidence, although there was also a significant relationship with the 
gamble score in the incentive condition. Confidence was significantly 
related to i nappropr i atness of Judgment and to willingness to take extreme 
positions under the incentive condition. That is, under the incentive, 
subjects who were confident about their responses were more willing to 
take extreme positions in responding. However, these extreme positions 
did not match their degrees of confidence very well. Several other of 
the i tern scores were related to confidence in the relaxed condition, while 
the gamble score became less important. The personality variables showed 
substantial 1 ntercor re.l at i ons. as did the cluster of gamble and appropr i atness 
scores. The significant negative correlations between the gamble and the 
appropriatness scores Is due to the fact that these two scores are not 
independent of one another. The negative sign becomes obvious when one 
examines the means of computation of the appropriatness score (WPLN) from 
gamble score (PLN). 

In order to better understand what variables contributed to ^'s 
expression of confidence, a regression analysis was performed. No sig- 
nificant regression held between confidence and any other variables, 
although high succorance and low harm avoidance did contribute a small 
amount to the prediction of confidence in the relaxed condition only. 

Seventeen of the scores were factor analyzed. The PLN and WPLN 
variables for the item of medium difficulty were left but since they did 
not seem to provide much information. A principal components analysis 
was first performed. Then the principal components were rotated according 
to the following specifications: A maximum of nine factors were to be 

extracted, the lower limit of eigenroots was set at 1.00 and no factors 
were to* have loadings of less than .30 for at least one variable. 

According to these specifications, seven factors were rotated. Ten rota- 
tions were required in the incentive condition. Thirteen were required 
in, the relaxed condition. The factor matrix is shown in Tables Hand 12 
Loadings in excess of 0.30 are underlined. 

In interpreting these results, it should be recalled that a low 
numerical score on the Appropriate variable means that a person's 
responses were congruent with his confidence. The factor analysis did 
not reveal much abput confidence, except to underline the fact that there 
is a dependence between it and the gamble and appropriate measures. This 
is illustrated in Factor 1 in both conditions. Factor 2 is made up of sex 
and several personality variables. Attitude is also a relevant variable 
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FACTOR MATRIX— RELAXED CONDITION 
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in the relaxed condition, but the importance of attitude in this factor 
is much reduced in the i ncent i ve, cond i t i on. Factor 3 in the relaxed 
condition and Factor 5 in the incentive condition are quite similar and 
are made up entirely of the personality factors. Factor 4 is perhaps the 
only one of much interest. It shows a relationship among sex and the way 
in which S^s deal with the difficult items. This, however, only confirms 
what has been previously said about sex differences with respect to 
making dogmatic choices on items. 

Cone 1 us i ons 



The findings may be summarized as follow: 

1. The reliability is proportional to the deviation of scoring functions 
from simple methods which subjects anticipate. 

2. The student is unfairly penalized by the scoring function which assigns 
him a score equal to the probability assigned to the right answer. Therefore, 
this function should only be used when minimal or no rewards are attached to 
subject performance. Otherwise the Euclidean function will not penalize 

the student, and will glye high reliabilities in a no-feedback situation. 

In the event that feedback through tables, computers, or scoring aids is 
available, the logarithmic function is recommended. 

3. On items not having unique correct responses, the weighted Euclidean func- 
tion is only slightly superior to the unweighted with respect to reliability. 

4. Under incentive conditions, scores on confidence tests are higher, 
and reliability significantly lower. 

5. Females have a greater tendency toward taking extreme positions than males, 
especially in the incentive condition. 

6. Subjects in the incentive group liked the test better, had more of a 
tendency to take extreme positions, and made more appropriate estimates of 
their confidence. 

7. Middle SES subjects, compared to both upper and lower SES subjects, 
made higher scores and more appropriate estimates of confidence. They 
seemed to be motivated more by desire for success than fear of failure. 

8. High scoring subjects gambled more on difficult items under the relaxed 
condition, but gambled less on difficult items in the incentive condition. 

9. Liking of tests was directly related to confidence. 

10. There was no significant regression between confidence and the battery 

of personality variables, although high succorance and low harm avoidance made 
small contributions to prediction. 

Much work remains to be done in studying confidence testing. 

Although it is clear that technical improvements may be made in the 
reliability and validity of tests through confidence scores, it is also 
clear that subjects do not handle their confidence uniformly. What is 
confidence to one may be hazard to another. As Wang and Stanley state, 

(1970) 
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The derivation of optimum response strategies in multiple 
choice test i ng represents an application oF mathematical 
decision theory which underscores the decision process 
inherent in such tests. The success of testing procedures 
which attempt to control the decision process will be 
critically dependent on the ability of subjects to effec- 
tively use optimal strategies. It is not certain that all 
subjects are equally capable of learning to use such strategies. 

The question of optimal strategies is likely to be perhaps the most 
significant outcome of further research on confidence testing. Although 
Bruner ( 1956 ) pointed out two basic differences in the way subjects use 
their confidences - the sentry condition and accuracy condition, and 
demonstrated empirical evidence of these two modes of behavior, there are 
other complex conditions which intervene between a subjective probability 
and a decision or action. Since it is possible, although not guaranteed 
that one may assess subjective probabilities accurately by means of 
reproducing scoring functions, two basic steps are needed. First, subjects 
need more experience in utilizing reproducing scoring functions. It 
takes a while to learn to respond intelligently to the rules of that game. 
Once it is possible to be confidence of measures of subjective probability 
on a set of subjects, further study may be made of the use of optimal 
strategies by subjects in problematic situations. Such strategies would 
perhaps start with what is known about optimal search procedures in 
polychotomic trees (Watanabe, 1969 ). Although the ability to utilize 
optimal strategies, and the ability to make appropriate assessments of 
one's subjective probabilities is of value in its own right, it would 
perhaps be a useful next step to begin to apply information about subjec- 
tive probabilities to the study of the structure of subject matter. This 
could be done through an analysis of the associati.ve networks of highly 
trained subjects terms utilizing a system of analysis similar to that of 
Quillian (1968), substituting subjective probability in place of his all 
or nothing at all lines of association. Further development of such 
techniques, and further gathering of data on sophisticated human subjects 
may lead to the uncovering of most of the appropriate parameters involved 
in guiding decision making in problematic situations. 

This goal is an ambitious one. Perhaps at a more realistic level 
would be the goal of increasing emphasis on the ways students react to 
problematic situations. Are students able to assess their state of 
information and respond intelligently to it? Do our teaching and testing 
practices make them aware that there are differences among the ways we 
use our information? 
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cc 

■"C ■ 

c 

"C ■■■■" 

c PROBABILISTIC TEST SCORING PROGRAM R„RIPPEY MARCH 1971 

■C" THIS PROGRAM ALLCWS THE SCORING BY ALL FUNC T I ONS ' I N ON E RUN 

C DECK SETUP 

' C CARD 1 IS A TITLE CARD ( COLUMN 2~ 72 

C CARD 2 IS A CONTROL CARD 

‘C"‘" COL 1 STANDARD DEVIATION OPTION 

C 1 ^ YES 

1C'” 2"= NO ■■ 

C COL 4 PUNCHED OUTPUT OPTION 

“C ~ 1" =" YES ■■ ^ 

C BLANK OTHERWISE 

"C COL 6 LOG' TRANSFORM OF EUCLIDEAN FUNCTION' 

C 1 = YES 

■' c ■ 2 = NO ■"■■■ :: " 

c NOT applicable IF EUCLIDEAN FUNCTION IS NOT REQUESTED 

■ C CARD 3 PARAMETER'CARO 

C COL 1 DATA TYPE 

C 1 = REGULAR DATA ‘7 

C 2 = PORTA PUNCH 

C COL 2-3 TOTAL NUMBER OF ITEMS ON TEST" 

C COL 4 RESPONSES PER ITEM — MUST BE SIX OR FEWER 

C COL 5-'6‘ NUMBER OF CARDS PER SUBJECT --"MUST '"BE TEN' OR FEWER 

C COL 7-3 NUMBER OF ITEMS PER CARD 

"C CARD' 4'FUNCTION SELECT' CARD 

C A 1 IS TO BE PUNCHED IN THE ^ COLUMN CORR ESPONDING TO THE- 

■ 'u" FUNCTIONS 'TO BE SELECTED' 

C COL 1 — EUCLIDEAN FUNCTION FOR SCORING INTRINSIC ITEM! 

■"C COL 2 -- SPHERICAL FUNCTION ' 

C COL 3 — TRUNCATED LOGARITHMIC 

■ C COL' 4 "— INFERRED choice 

C COL 5 — EUCLIDEAN FUNCTION FOR SCORING ITEMS WITH 

"C UNIQUE CRRECT ■ ANSWERS 

C COL 6 — SIMPLE PROBABILITY ASSIGNED TO Tj^E CORRECT AN' 

~“C'" ■ CQL-7 "--' ENTPdP'lC ■ 

C COL 8 — _ 

."C ■ coL"9'— "■ 

C COL 10 — 

"C KEY C'AROIST' "■ 

C FOR functions 1 ^NO _7_ . 

~ C ■ ■ PASS 'ONE OUTPUT 

C FOR FUNCTIONS 2, 3, 4t 5, NO 6 — - — 

■'C A SINGLE KEY CARD CONTA I N I NG 'THE' NUMBER'OF THE CaRritl 

C RESPONSES IN THE COLUMN CORRESPONDING TO THE NUMBER -r 

'C THE ITEM, FOR F.XAMPLEi IF THE' CORR ECT "R ESPONSE TO ^ 

C ITEM SEVEN WOULD BE RESPONSE TWO, THEN A 2 SH OULD Qc . . 

C ■■■ ''PONCHED' IN' COL'UMN 7, ■ ■ - 

C RESPONSE CARDS _ . ^ . 

"'C 'L AST' CARD OF E AC H JOB SHOULD ' BE ■ BL'AN'K . 

C JOBS MAY BE STACKED _ 

■ ‘C’ V LAST 'CARO OF LAST JOB 'SHOULO' CONT AI N 'A' - r'PUNCHED IN' CLS 1-2* 

C 1 — ■ 
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C OM MC M ?-1 ;V, VC , MO , K P , K I , K N f K K , K I C , i t A t 0 

dimension AVANS (72,6) , ANSPR ( 72 1 6 .1 t SI(72tlO), IO(lO», A' 
"i SS(72), S2(72)f i(72). AJO0(12), KS(lOJ, A(72»,3(72», 

2 Sr(lO), SSP(10JfREM(10J,VAREM(10),VAPP(lC),0EFK(10), 

"3 ■ CF( IO) ,SSI (10) , JFUNC( lOVt SM( 72T", D ( 6 ) 

C 

C' 

C 

C" 

c 

JOB = 0 

987 JOB = JOB + I 

WRITE OUTPUT TAPE 6t 986t JOB. 

FEAO INPUT TAPE 5t2235, ( A JOB ( I) , I =l . 12 ) 

■ WR I TE OUTPUT TAPE 6 , 2235 , ( A JOB ( I ) , I =1 I'' 

C 

c: 

c 

c 

c 

DO ^000 I = 1,72 
SS( I ) = 0 
S2( I) = u 
£(II = 0 
A( n = 0 
B( I I = 0 
DO AOOl J = 1,6 
AVANS(I,J) » 0 
ANSPP( I ,JI = 0 

4001 ANS(I,J) = 0 

DO 4000 J = 1,10 
4000 SI ( I , J) = 0 

DO 4002 I = 1,10 

SSI ( I ) = 0 
ssT( n = 0 
ST( n =0 
SSP( I 1=0 
REM( n = 0 
DEFP (11=0 
VAREM( n = 0 
VARPd ) = 0 
RFi ( n = 0 
CF( n = 0 

4002 JFUNCd ) = 0 
EN = 0 

BZRK = 0 

c" 

c 

c ' 
c 

C ■ 

READ INPUT TAPE 5 , 1 , MA , MB , MC , MD 
I F(MA)O40,940,241 

241 read INPUT TAPE 5,1, KF ,KI , KN, KK, KIC 

WP “IT E ■ out PiJ T T AP E ■6, 930 , K I , KN 

READ INPUT tape 5,4.050, ( JFUNC ( I) ,I =1 , 10) 

'c 

\t 



IS (7?, ; I 
. SST( 1 
R£L (1 



O 



. i \ 




c 

c 

c 

c 

IF UFUNCm + JFUNC(7» + JFIJNC(8H 4020t4020,402l 

4020 IF (JFUNC(2)+JFUNC(3) + jFUNC(4)+JFUNC(5r+JF(JMC(6M 4022, <.022, 4023 

4021 IF < JFU!MC(2)+JFUNC(3)+JFUNC(4)+JFUNC(5)+JFUNC(6) ) 4024,^.024.^025 
4023 WRITE OUTPUT TAPE 6,4030 

CALL SFARCri 
GO TO 987 

4023 GO TO 4 

4024 GO TO 3 

4025 WRITE OUTPUT TAPE 6,4031 
CALL SEARCH 

4040 GO TO 987 

3 KL=KI KN 
D082 K=1,KL 

READ INPUT TAPE 5 , 120,1, J,AK 
82 AVANS(I,J)=AK 
GOTO 6 

4 READ INPUT TAPE 5 ,994, ( I ) , I=1,KII 
DO 5 1=1, KI 

M=7 ( n 

5 AVANS( I,M)=1.0 

6 CONTINUE 
C 

c 

c 

c 

c 

IF (JFUNCISn 15,15,14 
14 DO 8 J = 1,KI 

8 B(J) = 0 

DO 7 I =T,KI 

DO 9 K = 1,6 

9 D(K) =0 

DO 10 K =1,6 

DO 11 J = 1,6 

COMP = AVANS(I,J» 2 

IF(K- J) 1^,12,13- 

12 CGMP = (1, “ AVANS(I,J)» *=♦ 2 

13 D(K» = D(K) + COMP 
11 CONTINUE 

10 D(K) = SORTF(D(Kn 
8(I> = D(l» 

DO 7 K = 2,KN 
IF (D(K) - B( I J) 7,7, 18 
18 B( I) = DIK) 

7 CONTINUE 
C 

C 

C 

c 

c 

WRITE OUTPUT TAPE 6, 5002 
5002 FORMAT I21H1MAXIMUM DISTANCE PORI 
DO 5000 I = 1,KI 

5000 WRITE UUTPUT TAPE 6, 5001, I, ( A VANS H , J ) , J=l , 3 ) , Bill 
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5001 FORMAT (5H ITEM, 15, 3H ♦ 3F10.4, 4H IS ., F10,4» 



15 CONTINUE 



211 GO TO ( 101,102 ),KF 
101 JP*1 
K»1 

IF (KK-1) 32,32f999 
999 00 121 K=1 ,KK 
JL=Ks^KIC 

IF ( JL-KI )A2,42,32 
32 JL=KI 
42 CONTINUE 

READ input TAPE 5 , 11 1 t IOC K ) , KS ( K ) , U ANS C I , J ) , J = 1 ,KN) , 1 = JP, ji. | 
IFC IDCK n940,240,121 
121 JP*JL+1 

GO TO 37 



102 JP=1 
Ksl 

IF (KK-1) 532,532,998 
998 00 172 KalfKK 
JLaK'JKIC 

IFC JL-KI >533,533,532 

532 JL=KI 

533 CONTINUE 

RFAO INPUT TAPE 5 , 1 12 , IOC K ) , KS C K ) , C C ANS C I , J ) , J= 1 , KN I , I=JP , JL I 
IFC lOCK) )940, 240, 1.72 
172 JP=JL+1 



37 G=IOCl) 

EN = EN+1, 

,N = EN 

IFCG)94b,240,4l 
41 CONTINUE 



IF(KK-l) 36,36,81 
81 CONTINUE 
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DO 36 K = 2,KK_ 

LA* I'D (K) 

LB=ID(K-1 ) 

IP (LA-LB)33f 80, 33 ' 

80 LC=KS(K) 

; ■ LD=KS(k-'i )■ 

IF(LC-LD) *5,35,36 
3'3 WRITE 'OUTPUT TAPE' 6,113,ir)(Kr 
CALL SEARCH 

GO TO 987 

35 WRITE OUTPUT TAPE 5,114,ID(K» 

■ CALL search’ 

GO TO 987 

36 CONTINUE' ' 

C 

'C 

c • . . 

'C 

c 

write OUTPUT TAPE 6, R003 

6,R004, ID(1) _ 

■c 

■'C ■■■■■■ 

c'.' 

" c ■ ■■■ — ■ - -■ — -- 

C PROGRAM NOW WEIGHTS AND COMPUTES PROBABILITIES 

DO 367 I=T,KI 

SM(n=0 

00'36'6 J'=1,'K'N"' ■ 

366 SMm = SM( I ) + ANSI I , J) 

8;PK = ' 'SM( I) ■■ ■ 

IF(B£RK)5327, 5327, 5328 

"5327 . ANSPRd ,'J) =' 0. 

GO TO 367 

532 8 CONTINUE 

DO 367 J=1,KN 

ANSPR(i,jr= ANS(i,'jr'7 SMTir 

367 CONTINUE 

■ "WR ITE OUTPUT TAP E ' 1 24 , " ( ( A N S P R Cl VJ r, J = 1','3'r, T='lVk I > 

C 

■'C'"' ■ '■■■■ 

C • 

"c " ■■ ■■ ■ " ■ 

c 

'■ '4061" IF(JFUNC(1» ) R062, A062,4071 ■ 

4062 IF(JFUNC(2)) 4063,4063,4072 

" ' 4063 ' ,IF( Jf'uNC ( 3) ) ■<♦064,4064,4073 

4064 IFIJFUNCIA)) 4065,4065,4074 
'4065 ~"T F ( J FUNC ('5') r'4066, 4066 ,4075 
4066 IF(JFUNC(6n 4067,4067,4076 
■"4067 '"IF( JFUNC ( 7 ) » 4068,4068,4077 
4068 IF(JFUNC(F)) 4069,4069,4078 

"'406 9 I F( JFUNC. (9 ) ) ^070,^070,4079' ■' ■■ 

4070 TFIJFUNCdO) ) 4090,4000,4080 
■■'4071 'C ALL FiJNC A ( AVAN'S', A"N S'PRTSS'pVST, SST',"S'T , S'S , STJ ^ 

GO TO 4062 

4072 ■■ calx FUNC^fAVANS, ANSPR ,SSP, si , SSr,STl 

I ..ERiC'-- ■ 





GO TC ^063 

A073 ■■ call FUNCC( A VANS ♦ A\SPR » SS^t SI ♦ SST', ST'T 



40 74 


GO TP 4064 

CALL FUNCCU AvA\S,AN'SPR ,SSP, SI,SST,ST) 
GO TO 4C65 






40 75 ■ 


CALL Fl)NJCF( AVANS, ANSPP ♦ SSP, SI ,SST,ST) 
GO TO 4066 






4076' 


CALL FUMCF ( AVANiS, ANSP^ , SSP , S I , SST ,STI 
GO TO 4067 




- 


4077 


CALL FUNCG( AVA\'S,A.*JSf’R,SSP,SI ,SST,ST) 
GO TO 4068 






4078 


CALL F UNC H ( A V A NS, A N S PP SS P , S I , SS T,, S T ) 






' 


GO TO 4069 






~4C‘79' 


CALL FUNCI ( AVANS, ANSPR ,SSP ,SI, SST,STT~ 
GO TO 4070 •' 






4080 


CALL FUNCJ(AVANS,ANSPP.,SSP,SI,SST,STF .. 


•5t ■ ; . -■ ■ 




■ 4090 


GO TO 211 







C 

. __ _c 












c 

c 


• ■ !■ ■ ' 








240 

C 


CONTINUE 








c ■ ■ ... : • . ■ ■ ■ ■■ •/ ■... 


c 

c 




V. 






c 


WRITE OUTPUT TAPE 6,4005 








c 

c 






■ J 




c 

c 











E K I = K I ■ . ■ I 

DO 4006 J = ItlO 

MB' ='■ j"' ■ 

IF (JFUNC(MB)) 4006, 4006» 3007 

c ■ •" ■ ■" 

c 

“3'007 ■■ G0“ TO ,^22 , 923 , 924,92 I ,2002V2004TVMB' 

921 WRITE OUTPUT TAPE 6,926 



- 


GO to 925 

922 WRITE OUTPUT TAPE 


6,9.27 ' 










> 


GO to 925 

923 WRITE OUTPUT TAPE 


6, 928 












GO TO 925 

924 WRITE OUTPUT TAPE 


6V300-2^ ■ 










} 

* 

^ I . 


GO TO 925 

2002 WRITE OUTPUT TAPE 


6 ,2003 ■ . : ■ ■ 










f 


GO TO 925 

2004 WRITE OUTPUT TAPE 


6,2005 




' 








925 CONtiNUE 

C ^ 














C 












f > . 


CF(J) = (STU» ! 


ST( J») / (EKI St EN) 








: . 


i 

iFRir 


SST(J) = SST(J» - 


cF(ji r:”" 











1 ST:;- 






702 



705 

'706 

636 



A006 



DO 702 I = 1 f KI 
SSI ( J > = SSI f J ) "+■■ 
SSKJ) = (<SSI(J» 
SSP( j). = ( ( SSP ( j) ) 
DO 705 I = 1 ,KI 
SI ( I ,j > '= sniVjV 
WRITE O’JTPIIT TAPE 
FORMAT! 13H0 ITEM D I F 
DO 636 1=1, KI 
WRITE OUTPUT TAPE" 
REM(JJ = SST(J» - 
"D£ FR < J.) ■■■=■■■ (■£ 
VAREM(J) = FEMiJ) 

VAr p ( J ) = "s S P ( J ) 7 
REL(J) = 1. - (VAR 
"WRITE OUTPUT TAPE" 
WRITE OUTPUT TAPE 
SST( j) , CF("j) ,ST( J 
CONTINUE . 



s‘in,“jr‘^ srnv 

/ EN» - CF( J) > 

V EKI r - CF ( J) 



/”EN’‘"'."“" 

6,706 

FTCOLTlESi 



xr 






6 :; 70 av 17 srnv 

SSP(J) - SSKJ) 

(EM'-'Cix.r- 

/ DEFR(J) 

■■■( EN-''--r, f " . 

EM(J) / VARP(J) 
6,7.42, REL( J)7 
6,776, VARP(J), 
J" 



J) 



VAREM( J) ,REM( J),SSI( J) , SSPCJ), 



C 

C 

C 

C 

"c 



MA 



773 



25 



940 



GO TO (987 ,773 ) 

"EN"= E'N-1 •■■■■;" -7: , . 

WRITE OUTPUT TAPE 6',628 . _ . ' 

'DO"25 ■"r=17'KT"'"".'7-----7“- -——--- 7 ;-— 

S S ( I ) = SS ( I )"‘ SS ( I) /EN - 

SS( I )= SORTFt (S2( I )-SS( I) )7(■6^^-r•"^f7 
WRITE OUTPUT TAPE6,125,I,SS(I)' 

" GO ".fcTgsT- ■ ■ 7 , •" ■■■; , ; 

CALL EXIT ■ ■ ■■ 



C 

c 

e' 

c 



1 

'll 1 ' 
112 

113 

114 
116 
124 
1 18 
119 
1 2 Oi 
122_ 
12 '5 
628 
■ 704 ' 

742 
'776 
926 
'927' 
92 8 
929 



FORMAT! 
''FORMAT 
FORMAT! 
FORMAT ( 
FORMAT! 
'FORMAT!' 
FORMAT! 
FORMATi 
FORMAT 
FORMAT! 
FORMAT! 
■FO'r'MAT'! 
FORMAT ! 
■■ FO'RMAT 
FORMAT 
FORMAT 
FORMAT (_ 
"FORMAT"! 
FORMAT! 
FORMAT! 



11,12,11,12,12) 

TF5 , 'i 1 , i6T3F'i'7o""' )')" 

I5,I1,36F2.0) 

3 5H "E R R OR ' ' IN' ' I D " 'C 00 E S' "FOR 
39H CARDS CUT OF SEQUENCE 
6 TI X , 1"2 , 1 H- ;T2 , 1 X VF 5 VE ) 'T 
12! 1X,F5.2),I 

14H TOTAL "S'COR E' F ll* 2 7$X , 1 $') 

!/16H ITEM SCOPES FOR , 2X, .15.) 
IX , I 2, IX ,12 , IX, p 5 - 21 ::; :-; ■-•■•----- 
IX, F 2 ; 0 , 1 X ♦ F2 • 0 , 1 X ,F 5 ^ 2) 



"SUB'JECT"‘NOV 
FOR SUBJECT 



"715) 

NO. 



15 1 



!2,2X,F7.2) . , ■ ■■ ■ ' 

50H CRITERION GROUP 'STANDARD DEVIATIONS 
■"( 2 xV' 1 2 ,::3x77 572 r""""' v^' 7 ,, 4. 1 

(23HORELI.ABILTTY OF TEST = ,F6,3,2X,5H FOR ,I4_i2^^ 

"! o7-ixVF772r)',"'"'""'""""”""'^""'"^"''''""7'". "■"""""■''""■T:.-.: .v.-;: 

27 HI E.UCL I DE AN SCORING FUNCTI ON) 7 ' ' - 

' 2'7HrSPHERlCA'L SCO'R ING FUNCTTON T ^..77^ v 

27H1L0GAR ITHM SCORING FUNCTION) * V 

■39hireproducing euclidean‘scori n 6 FTiN'c fr ONrr"" ■" 



"'.-.trs- 



o ■ 

ERIC 



SUBPOUTI N5 SEARCH 

SURPOUTINF SEARCH _ 

2 READ INPUT TAd£ 5fl» ID 
1 format (15) 

IF( ID.)‘ 9?Q,3,2 ‘ 

3 RETURN 

9 99 ■ CALL EXIT 

end 



• )t:.- ■ - 




c 

c 



SURROUTIN: OUT(FT) 

(:oMNON 'MA ,Mr , NOrkF V KI » KNt' KKV«< rC ' 

DIMENSION £<?2)t SF(72). A(72» 

EkN=KN ■ "" " V 

EKI=KI 

A = EkT/i2V — r— - 

KA8=kl/l2 ■ ■ ■ 

EkAR«XAB'-''' ■■■ “ -r--————-.- 

IF(EKA3-AI2C€V207t206 
206 k7P=‘KA8+l' ■■■ 

GO TO 208 
207"kTp=kAB — 

208 CONTINUE 

JL = i "■■■' ' ■■ " 

DO 209 IK=1 ,KI(> 

"■jM=12^Tk“~ ■ ■■ 

IF(MCJ232t2?2t233 

^233"‘PUNGH 124 , ( i ( I T, I sJLVJMI' ' " 7 — — 

232 WRITE OUTPUT TAPE 6 , 124, ( Z (I) , I=JL , IM) 

" 2 09 " JL =T 2*~i K " ♦ 1 ■ — — - ------T---- --- 

!F (MC)254,234,235 
2"35 pu^^cH ir9,'' F;r 
234 WRITE OUTPUT TAPE 6V118, FT. 

'1 24r F0RMAt ( i 2 ( 1 X , F5 • 2 ) )"" — 

lie F0RMAT(14H TOTAL SCOR E = ,F7.2 , 5X, 15 ) 

■return"" ! — — — 

END 



o o', 00:0 



930 FOPMAT (2Xf I3f 1X,6H I TEMS t IX f 1 1 f 1 X f 19H RESPONSES PER ITEMI 

";■■■ 933 FORMAT(lx;ilfI2iIlfI2»I2.40XV F6T0) 

986 FOPMATdlHlJOB NUMBERflXfiar 

( 72 F! .'Q') ■■■ r- . 

2003 P0PMATH7H1PR0BABIL1TY ONLYI 

2005 fORM‘AT'(i 8 Hl ENTfi CPIC FUNCT I ONT ' "7 . 

2235 F0BMAT(12A6> • 1' ^ . • '■ - 

■ ‘'SOO^ F0RMAt (16'HllNFERRE0' CH0lCE I . ' 

4003 FORMAT ( 1H1/50X ,20H— STUDENT SCORES —I , 

"4004 ‘ FORMAT { 15HOSTUOENT"NUMBERV ’AX'i ‘ TS") " — -7-— — - 

4005 FORMAT (1H1/39X,A2H — ITEM OiFFICULTIES AND REjL I ABIL I T1£S — » 

“"■‘"■'4630 ‘FORMAT t 30HCNO SCOPING FUNCT ION REQUESTED) ---r— - 

4031 FORMAT (41H0INC0MPATIBLE SCORING FUNCTIONS REQUESTED) 

40'5D' PORMArdOTl) - -- 



SUnPOUTINF AVAMSt A'lSPR t SS^, GI » SST ,_S_T , S_S , S2 > 

COMMON MAfMC .Mr'.K.c,'-;! ,r<r,,KK,KlCtivSFt A 

DIMFNSION AVANS( 72 , AN S PR ( 72 1 6 ) t SI(72tlO), SST(IO), ST(10|, 

1 ■ SS (72 ) , S2(72 I , S57(10/t Z ( 72 ) t SF(72)t A(72» 

201 DO 203 1=1, KI 

SF( n=o ■ ■ '■ 

FT = 0 

203 Z( 11 = 0 ■ - — 

DO 20^ !=1,KI 

."■do 295 J = 1,KN " 

295 SF(I)= ( AVANS(T,J» - ANSP?(I,jn ^*2 + SF(I) 

Z(I)= 1.- ( SOFTF( SF( I) » ) / 1.A14, 

.IF (MD-1) 1112, 1112,1111 

1111 ’ iFiZd) - ,0U 1113,1113,1114 

1113 Z(I»=0. 

^G'O ■T0'"1112 '“" •• 

1114 Z(I»= (2. ♦ (LOGF(Z(n)l / 2-30261 / 2- 

1112 SKI, II = S!(I,l) ♦ Z(II - - -- 

SST( 1 ) = SST(1 I Z( I I ^ Z( I I 

ST(!I = ST(l) * lH) ■ ■ ■ 1 ■ 

GO TO ( 204, 9) ,MA 

~9 "ss( I )= ss( I j( 1 1 ■ ' ' ' ■ : “ . - - 

S2( ! 1= S2 (I 1+ Uli^H II 

■"2C'4' Ft=FT-^/( I I ■ ' 

WRITE OUTPUT TAPE 6,1 

i" FORMAT (//lOHOEUCLIOEANr ■■ " : 

CALL OUT(FTI 

■■ “SSPill = SSP(l) V FT > Ft " ■ ■' 

RETURN 

END 



C 

subroutine FllNC3( AVANS, A.NSPP ,SSP, si ,SST,STI 
■" 'C OMMON .M A , •1C , MD , KP , K I , K N, KK , K I C , Z , OE N2VAMUM— 
DIMENSION AVANS (72 ,5 I , ANSPR(72,5I, SI(72,101 
1 SSPdO), Z(72), DtN2(72), ANU;M(72r ^ 

202 no 214 1 = 1 ,KI 

■z(ii=o ■ ■ 

D5.N2( I 1 = 0 

'ANUM( I 1=0 " ' 



214 FT=0 

217 DO 215 I=1,K1 

DO 216 J=! ,KN 

DEN2( I l = nr’.‘.?( I ) +■ 

216 ANUM( I l = ( AVAV'ST ! ,J ) 



( ANSPR ( lVJ rf«‘=>2 “ 

I ^ ANSPR ( 1 , JI+ ANUM( K 



DEN2(II= S TP(DFM2( I ) ) 



z( I i = AN<jM( : )/nEN2( 1 1 

SI (1,21 = S!C,2I + Z(T1 

SST(2) = SST(2) ♦ £(n Z(II . 
ST(2 I = ST (2 ) ♦ 7( I I 
215 FT = PT*Z( I I 

■SSP(?) = SSP( 2 1 ♦ PT FT 
WRITE OlJTi^ijT TAPE 6,1 
1 FDR m at ( / /I C HC S PHE R I C AL I 
CALL OUT(FT| 

RETURN ^ 

END 



SSTdOl, STdOl, 





C 

SUBROUTINE FUNCC I AVANS , ANSP!^ t SSPt S I ,SST,STI 

■ COMMON MA,MCVmD,KF,KI ,KN,KK fKICf2fSFVA 

DIMENSION AVANS<72,6», ANSPM72,6I* SH72,1CK SSTIIOI, STIIOI, 

1 SSP(IO) , 2(721 ,SF( 72r,A(72r ^ 

205 FT=0 

■ DO 614 'i=r,Ki' ■ 

sF(n=o 



'■■■6l4'2lT 1=0 ■■■ 

DO 615 I=lfKI 

DO "616 j=i V kN" '" ■■ : v 

616 SF(n» AVANS(I,JI ^ANSPP(I,JI +SFm 

IF (SFm-. Cl) '617,517,618 ! ^ 

617 2(11=0 

GO TO' 621 

618 2 (I»* ( 2 . •MLOGF(SF( n n/ 2 . 30261 / 2 . 

'“"621 : CONTrNUE' ■■■ ■ ■■ ■ " 

SKI, 3) » SKI, 31 2(11 

' SST( 3 i =" SST( 3 1 + 2(1 j i 

STOI » ST(3I + 2( n 
■ '615 FT« Ff+' 2 ( I ) - . 

SSP(3I = SSP(3 1 + FT FT 

WRITE OUTPUT ■ T APE 6 1 " 

1 FORMAT (//12H0L0GAPITHMICI ' 

■■■“■ CALL OUT f FT) 

RETURN 

■■■* •" "END ■ ■■ — 










mm rm» 

DO 



c 

c' 

c 

'c' 



SUBROUTINE FUNCO ( AVANS t AN SPR t SSP t S I » S ST t ST ) 

COMMON MA,MC,Mn,kF,Kl,KN;KKfKICt ?iAfB ' ■ ' -■ 

DIMENSION AVANSr ^2 f 6 ) f ANSPR(?2t6)f SI(72tlC), SSTdOl, STdCI, 
1 " SSP{ 10) t d 72) f A{72rtBT72 ' " ' ’ 

271 FT=0 

DO '301 O ' ■ I=l’, KI ^ ——— -7 ■ 

3010 Z( I )=0 „ . 

1=1", Kl". '" “ “ 

K=0 

■db ‘302 0‘ J=i%‘KN' ■ 

IF ( AVANS ( I , J) ) 



3020,3020,3021 



3021 

3020 



K=J 

CONTINUE 

ALTr'i) ■ 

00 3022 J=1,KN 

I'F (ANSPR‘ dVJT-'Al 

302A AL=ANSPP( I , J) 



GO TO 3022 __ 

■■■,M="b' — ■ — 

CONTINUE 

1 F {'M- k ) 30 2 5 7ro'26 , 

Z(I) =0 

■■GO' tO "3C4o 
Z( I ) = 1.0 



) 3022,3029,3024 



3029 

3022 



3025 



3025 






3026 

3040' 



CONTINUE - 

SI n L^i = s n I ,_4 )_j»- li ij ' 

~SST(4) = SST(4) + ZH) >5 Zd) 
ST(^) = ST(4) + Z( I ) 

'■ Ft = FT+Z( I ) : 

SSP14) = SSP(4) + FT * FT 
W R 1 1 E OU T P U T T AP'E' '6 ; f ". v ' ^ "" 

FORMAT { //16H0INFERRE0 CHOICE) 
'call 'out (FT)' " ■ ■ ■■ 



3041 

1 





END . 




■ • ■■■ . ■ . 


' \ 


y: V-. ; 












.• * ■. 
























<7-.' 


• v'-- 




‘ ^ 























O 

ERIC 



---"-^— 56 - 



c 


SIJRPOUTPIE Fij jCF ( AV ANSf ANSPB f SSPf SI f SSTf ST ) 








COMr'ON MA, xc, ’Of KF ,KI f KNt Kk tKIC . Z f SF » A 
DIMPf>jSin\i AVANS(72 1-6) f ANSPR(72t6)f SI ( 72 1 10), 


SSTdO ), 


ST (10), 


1 

2001 


SSP(IO), Z(72)t SF(72)» A(72) 
FT = 0 








DO 2074 1=1, KI 
SF(I)=0 






. 207A 


Z(I.)=0 

DO 2075 I=1,KI 






2076 


DO 2076 J = l,kN‘ 

SF(I)= AVANS( I , J)^ ANSPP( I, .+ SF ( I) 






Z( 1 )=‘SF( I 1 ' 

SI ( I ,6» = SI( I f6 » + I) 


SST(6l = SST(6) + /( i ) ^ Z( n 
ST(6) = ST(6) + Z( I) 


2075 


FT=Z(I ) + ft 

SSP(6I = SSP(6» + FT FT 






1 


WRITE OUTPUT TAPE 5,1 

FORMAT (//43H0PR0QA3ILITY ASSIGNED TO THE CORRECT ANSWER 1 


• 


CALL OUT (FT) 
RETURN 




* 




END ■. ■ 






C 








C 

c ■ 




SUBROUTINE FUNCE ( AVANS , ANSPR , SSP, SI ,SST,ST» 








common MA,MC,MD,kF,l<I,KN,KKfKIC,Zf SF,.A 
DIMENSION AVANS(72,5), ANSPR(72,6), 51(72,10), 


SST(IO), 


ST(IO), 


1 


"SSP( 10) ; Z(72) , SF(72), A(72T" 
DO 203 I = 1,KI 








SFm' = 0 " ~ 

FT = 0 






2 03 


z( I ) ■= 0 

00 204 I = 1 ,KI 






295 


DO 295 J = 1 ,KN 

SF(I) = (AVANS(!,J) - ANSPR ( I , J ) )=^*2 + SF(I) 








Z(I) = i; - (SORTF(SF( I) ) ) / 1.414 
IF (MO - 1) 1112,1112, 1,111 


r- 




" llil 

1113 


IF (Z(I) - .01) 1113,1113,1114 
Z(I) = 0 


• ' ” ' 




1114 


GO TO" 1112' 

Z(I) = (2. + (LOGF(Z(I»)) / 2.3026) / 2. 






1112 


CONTINUE 

SKI, 5) = SKI, 5) Z(I) 


9 






SST(5) = SST(5) + Z(I) ^ Z(I") 
ST(5) = ST( 5) + Z( I 1 






2 04 
1 


FT = FT + Z(I ) 

S5P(5) = SSP(5 ) + FT FT 

■'WRITE' 0UTPUT' TAPE'6’,r 

FORMAT (//ITHOEUCLIC. :N UNIQUE) 


— 





CALL OUT (FT » , . 

RETURN 



O 

ERIC 



C ' S8. 

C 

c - 

SUSP GUT I NE PiJNCjC AV4NS,ANSPR,SSP, SI ,SST ,Sn 

COMMON MA,MC,MD,KF,KI,KN,<K,KIC, Z.BL.A ■ 

dimension AVArJS(72,S» , ANSPR(72,5I , SI (72, 10), SST(IO), ST (10), 

■ 1 SSP(IO), ?(72), BL(72), A(721" 

2005 FT = 0 

DO 20C7 I = 1,KI • - ■ - - 

Z ( I ) = 0 

2007 BL( I ) = 0 - 

DO 2017 I = 1,KI 

no 20 CP j = ■l,KN : 

IF ( ANSPR ( I , J) ) 2010,2010,2012 

2010 BL(I) = -9999. 

GO TO 2013 

2012 BL(I) = LOGF( ANSPR (! , J)/2.3026) ' 

2013 DL(I) = AVANS(I,J) B(.(II 

2000 Z(I) = Z(I) + BL(I) 

Z( I ) = 10. + Z( I ) 

■ IF (nm 2016,2016,2018 " 

2016 Z( I) = 0 

2018 CONTINUE ‘ ' ' 

_ . SI( 1,7) = SKI, 7) + Z(I) 

SST(7) = SST(7) -f Z(I) > /(I) 

ST(7) = ST(7) -f Z(I) 

2017 FT = FT -f "ZCI )■'" 

SSP(7) = SSP(7) + FT FT 

WRITE OUTPUT TAPE 6,1 

1 FORMAT (//9H0ENTP0PIC) 

CALL OUT (FT)" 

RETURN 

END 

c “ — — 

_ C . . . ■ _ _ ' _ 

SUBROUTINE PUNCH ( AV ANS , ANSPR , SSP , S I , SST , ST | 

common MA, MC, •1D,KF ,KI',‘<N, KK,KIC, Z, SF, DMAX " 

DIMENSION AVANS(72,6.) , ANSPP(72,6), SI(72,10), SST(IO), ST(IO). 
~1 SSP(IC), Z(72), SF(72), 0MAX(72) " " 

201 DO 203 I = 1,KI 

SF( I ) = 0 

FT = 0 

2 03 Z( I ) ■ = ■'0 

DO 20A I = 1 ,KI _ 

DO 29? J - 1,KN “ 

295 SF(I) = (AVA\'S(I,J) - ANSPR(I,J)) » + SF ( I ) 

3C(i) = SQ3TP(SP(I)| ■■ ■ • 

Z(I) = 1. - ((SF(I)/OMAX( D) 2.0) 

SI (1,8) = SKI,?) f Z(il : 

SST(8) = SST(8) -f Z(i) 2(1) 

■ ST(8) = Sr(8) Z(I)' 

?0A FT = FT ♦ Z( I I . 

' ■■ write OUTPUT TAPS' 6, "26? " 

202 FORMAT ( //23H0EUCL IDEAN PRUPCRTIONALI 

■ CALL OUT (FT) 

SSP(8) = SSP(3) 4 FT FT 

RETUrN ■ ■■■ " 

E'j'' t.; 



oooo oonO 



SUBfiOUTINE FUNGI ( AVANS tANSPP , SSP, SI tSST ,ST » 

COMMON MA,MOtMD,KF,KI, KN,KK,KIC, Z,A,B 

DIMENSION AVANS(72,6)t ANSPR(72,6I, SI(72,10), SSTdOl, 
1 SSP(IO), i(72), AI72), B(72"» 

RETURN 

END 



SUBROUTINE FUNCJ ( AVANS tANSPR , SSPt SI tSST, ST» 

COMMON MA,MC*MD,KF,KI,KN,KKtKICt itA,B 

DIMENSION AVANS172t6»v ANSPR ( 72 , 6 )! , SI ( 72 , 10) , SST ( 10 ) t 
1 SSPdO), Z(72), A (72), 8(72) 

RETURN 

END 




ST(IO) , 



STIlO) , 
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’* XEQ ' • ■" ■ 

,* _ FORTRAN 

* BINARY ' ■ " 

C WEIGHTED EUCLIDEAN SCORING. ROBERT RIPPEY JUNE 1970 32 ITEM 

■C~^ THIS PROGRAM SCORES TESTS USING THE WEIGHTED OR UNWEIG"HfED ' 

C _ _ EUCLIDEAN FUNCTION .32 ITEMS MAX. ,3 RESPONSES PER ITEM AND A 

C" confidence' ME ASUR'EV fW0"XfNp"r"0r'K¥Y"S‘"MAY'‘BE"'U ' 

C THE FIRST CARD CONTAINSTHE FOLLOWING PARAMETERS. 

C" " " COL 1 KK ■ KEYfNT'OP'f fUN""""! R E-GUT AR","2 PHA'SE" r"OUrp-UT: 

C _COL2 PUNCHED OUTPUT IF 1, OTHERWISE BLANK. 

C COL “3 wriGHflNG OPTION 1 NO WEIGHTING, OTHERWISE BLANK. 

C COL 4 = KC, IF 2 IN COL. THERE ARE MORE THAN 16 ITEMS. 

C" Cdr'5-6‘''=klV NUMBXR'"d'F"lTEM5r"SdMF"NUMBER'F 

C. OPTION 1 CONTAINS KEYED ANSWERS IN APPROPRIATE COLUMNS 

opTTdN''2' utiL'riEs"TH£'OupuT""Dr''p'HA‘s"E''r : — 

C CARD ORDER CONTROL ,K EY S , T I TLE , RE SPONSES , BLANK , REPEATS,-! 

C RESPONSES" ^ 

C COL 1-6 l.D. CUL 7-70 RESPONSES UP TO 16 ITEMS IN GROUPS OF 4 

■"C ""The" FOURTH RESPONSE”"REPRESENTS'CdNFTDEN'Ce'r~0="NO"i:0NF I0tNCE^ 
C • 9 = ABSOLUTE CERTAINTY. COLUM 71 REPRESENTS TEST ATTITUDE. 

"c "d" ="hate the test. '9"'= ■'love"Th'£' 'fesT. : 

C THE SECOND CARD CONTAINS RESPONSES FOR ITEMS 17 ON 

C TITLE CARD CONTA INS TEXT-’IN "COLS"r^72 ^ 

C THE FORMAT OF THE SECOND CARD PER SUBJECT IS THE SAME AS ‘ CARO 

r END ■ OF JOB ■ blank' CAR O' 

C END OF RUN -1 IN COLS 1-2 

"i D I MS N S I ON AT I T L ( 1 2 ) A'RFY (1273 Y 32,3 T,'"' 

2PANS(32,3) ,CON (32), IT(32), DMAX(32), D(32) , S(32),AT(32) 

“ 3 ", ZV(32,300) 72(3 m S ST3'2T , — S P T3 0 DVT ^STr3*2’l 

EQUIVALcNCc ( DEn2 , S2 ) , ( ANUM , SS ) , (Z,S) ,(SK,FT) 

'999 ■ HE AD ( 6 , 99 3 ) KK , MPUN , MWT , KC','i<T " ' 

993 FORMAT (^11, 12) 

GO TO (992,991),KK “ 

991 KL=3*KI 

DO '82 K = 1','KL' : 

READ(5, 120) , I ,J,AK 

82"AKEY( 1,J)=AK " “ " ^ 

GO TO 707 

1 20" FOR M A T (1 X ,' 1 27 i'x7'f 27 Vx" rF’5"."2T ■ 

992 READ (5,1) T 1 D , ( { AKE Y ( I , J ) , J = 1,3 ),I = 1,16) 

■ 52 F0RMAT(4( 2X,F6.2)‘) ^ 

1 FORMAT( F6.0,16(3F1.0,1X) ) 

IF (KC-1 >800,800,80 1 ^ " 

801 READ (5,1) TID,((AKEY(I,J),J= 1,3 ),1= 17, KI ) 

"■"8 0 0 ■ r 0 N T I N U £ " 

707 CONTINUE 

IF (T ID) 2,99^7995 

995 READ(5,5) ( AT I TL ( I ) , I = 1 , 1 2 ) 

5 ' FORMAT ( 12A6) ' " " 

0=0 

DA = 0 " “ 

DB=0 

~ SO B = 0 ^ 

DC = 0 

£N=0 ■ 

TLIK =0 

EKI=KI 

S.SP=0 

CF=0 

I--' ST = 0 : 



ERIC 



sTi=o ■ 

SSP = 0 

REM =0 

DEFR=0 

"VAREM i'O " 

VARP=0 

REL "»6 

FT«0 

Od' 8 ‘ I=Y f KI 
SI(I)»0 



iT( n*o 
CON ( I)=0 

SiDaQ 

00 555 N«lf3dO 

-55-5- zv ( I j »d' " 

DO 8 J>It.3 
pk e y V i Vj ) 

ANS( If J)^0 
’■“■PANSI I f J ) *b‘ 

8 CONTINUE 

■ WRITE (6 rs) '(ATTfrTf)7i«‘rfTrr 
WRITE (6f6) TID 

6 FORMAT! 13h~TEST"NUM6ER fF6.0) 
DO 7 I * I f K I 

"00" 7 J»l f'3 

7 ATI I ) * AKEYI If J) + ATI I) 

- IP ( Ati i )) 5 ddf sod ; 501 

500 WRITE (6f 502) 

502 FORMAT ( lOH KEY 'ERROR') 

CALL EXIT 
"501“ "DO 9 I=lf ki 
DO 9 J=lf 3 

- 9 -pKEY ( i , j r * A'K6'Yri7yf-7-A“TTIT 
DO 16 I*lfKI 



CALL MAX ( PKEY ( I V I ) f P'KE YTI f 2 ) > PKEY II f 3 ) f DM'AXTTT) 

WRITE I6f5lO)fPKEYIIfl)fPKEYIIf2)f PKEY Hf3)f DMAXII) 
“■5iO‘"‘FORMAT 1 4 i F6. 3 , 2X )') ! 

16 CONTINUE 

yo"*RE AD 1 5 f 20 )"sTD7”rr7rNsr i73TrJ^^ i7yr;roTj“rr)7i"= 171'^^ i k 
20 FORMATIF 6 .O f65Fl.O) 

~IF ikc-l >803f 803f804 

804 read I5f20) SOBf I I ANS I I f J ) f J = 1 f 3 ) f CON II)fI = l7fKl) 

"'863 ' CONTINUE ' 

TLIK = TLIK + ALIK 
"SK’O 

IFISIO) 2f240f842 

“nB'42“EN = t N + 1 * 0 ■ 

N=EN 

"00 21 I = lf'KI 
DII)=0 

^1"'ATI I)=0 ■ 

00506 I=lfKI 



22 

_ 50_4 

23 



O 

ERIC 



UO 22 ’ J = lf 3 

ATI I ) = ANSI I f J) *■ ATI I ) 

""IFI ATI I ) r'50375037504 
CONTINUE 

150 23'J = li3' '■■■ 

PANSIIfJ) = ANSIIfJ) / ATII) 

-crrrr’=o ^ 

DO 25 J=lf3 



61 



25 D(I) = (PKEY(I,J) - PAI\JS(i,J))»*2 + D(I) 

D( I ) = SORTF(D( 1 ) ) 

S(I)= 1.0-((D(I)/DMAX(I)) *2.0) 

IF (MWT)24.2A.5U 
2A 5(1) = S( 1 ) * C0N( I ) 

50 SK = S( I ) + SK 

SCON * SCON + CUN (I) 

ZV( I,N)= Z( I ) 

GO TO 506 
503 S( I )=0 

ZV(I,N) =0 
506 CONTINUE 

SSP=SSP + FT*FT 
AVCON - SCON / EKI 
WRITE (6f 2 nSIDfSkf ALIK 

27 F0RMAT(8H STUDENTfFB.OfllH SCORE = ,F7.3,7HATT= ,F6.3) 
WRITE (6,67) AVCON' 

67 F0RMAI(22H AVERAGE CONFIDENCE = ,F5.2) 

SCON » 0 

26 FORMAT! 16( 1X,F5.2) ) 

■WRITE (6,26) , (CON( I ) , 1 = 1, KI) 

32 FORMAT (IH ) 

WRITE (6,26) ,(S(I),i=l,Kl) 

IF (MPU'N) 60,60,31 
60 WRITE (6,32) 

GO TO 3 

31 PUNCH 52, SID, SK, ALIK, AVCON 
PUNCH 51, (S(I), 1=1,16) 

iF(KC-l)810,810,811 
811 PUNCH 51, (S( I ) .1=17,32) 

810 CONTINUE 

51 FORMAT! 16F5.2) 

3 CONTINUE 

GO TO 30 

■ 2^40 DO 701 I = l,Kr 
DO 701 J=1,N 

s K I ) ■= “S i { I )■ +“ ZV^nV3T 

SST= SST + ZV(I,J) * ZV(I,J) 

•~TDT'1ST= ‘ST" "+"7V ( 1 , j )■ “ 

CF » (ST*ST)/ (EKI*E N) 

^SST=SST-CF 

DO 702 1=1, KI 

"To?”ssr=ssi + SI ( I )*si nr 

SSI=((SS1)VEN)-CF 
SSP = ( (SSP)/EKI)-CT 
DO 705 I=1,KI 
~T05~ST{I)=SI(I)/EN 
AVLIK = TL IK/EN 
WRITE ( 6,537) AVLIIC 
537 F0RMAT(18H AVERAGE LIKING = ,F5.2) 

WRITE OUTPUT TAPE 6,706 
706 FORMAT! 18H0ITEM DIFFICULTIES) 

DO 636 1=1, KI 

636 WRITE OUTPUT TAPE 6 , 704 , I , S I ( I ) 

■■704 FORMAT (2X,I2,3X, F5.‘2) 

REM = SST - SSP SSI 
— defr = (EKi-1. ) ■*• (EN - i;t 
VAKEM = REM/DEFR 
~VARP =-SSP/(EN -ITT 

O REL = I.-IVAREM /,,VARP) 02 

ERIC Li 



C 

n 

C ■■ 



c ■ 



C;. 

o' 



o*._ 

•) 

G ■' 



c 













W R I T E " WtP U f ■ ■ T AP E'“677 A2TREX"7I^" ' " 

742 FORMAK 2 3HOREL 1 AB I L I TY OF TEST = ,F6.3,2X,5H FOR ,14,2X, 4H SUB> 

"write output tape '7/7 76V varp7v"ar¥m^^^ ^^"rem7ss~IV"sspV"s"sT;^^^^ 

776 FORMAr(B( IX, F9.D) 

i F ( S 06 )■ "2 , 9 99, 999 ^ : 

125 FORMAU I2,2X,F7.2) 

■' ^""CALL' exit ^ " 

END 

SUBROU i riVE' ■ M'AXX PA ,"PBV PCTDMA'F")" " 

DMAX =0 

DA='0 ^ 

DB = 0 

DC = 0 ■ 

DA = SURTF( ( 1.0“PA)**2 + PB*»2 + PC**2 ) 

■"DB" = SQRTF ( PA**2 + '( I /0-P8 )'**2“ +'" ' — " 

DC= SQRTF ( PA**2+P6**2+ (1.0-PC)**2) 

IF ro A -- D B ) 1 0 , 11 n ^ 

11 DMAX = DA 

GO TU 12 

10 UMAX = DB 

■ ■ 1 2" ■ IF (DMAX -DC' ' ) ' '137 1 AV 1 4 “ 

13 DMAX = DC 

GO' TO 15 T" ^ 

1.4 DMAX = DMAX 

“1-5 ■ CUNT IinIUE " 

RETURN 





^ J 



63 



( 



(. 

1 




6 . 



C;. 

3 . 

C 

9 . 

Cu. 

c 



C:. 

c 






* ‘ XEQ 

* FORTRAN 

V fcTlN'ARY 

C RISK TAKING INDEXES AN NE MARIE GROZELIER M AR CH 70 

D I MtNS I ON A ( r& )7 b (1 5 ) , C ( 1 5 ) , D ( 1 5 ) , pin ( 1 5 ) , RT ( 1 5 ) 

D^99£]TPLN=0 

TRt = 0" 

TC = 0 ' _ 

RE A D77;Trro7u fiTrB'rf ) vcTiT, □ 1 1 > , f= i , 1 5 1 

1 FORMAT! 16, 6 OFI.O) 

I F ( Tb ) 9d , 98 , 5‘ 

5 DO 2 1=1, 15 

■pLNnT=T( A( i )"-37b)**2''+(B( I )-3.0l**2+(C( fl -3 . 0 ) **21 /6. 0 
RT(I)= ABSF(PLN( I )-D( 1 ) ) 

t R f= t R fV R t ( f) """ 

TPL(M = TPLN + PLNiI ) 

Tc=Tc+ori r 

2 CONTINUE • _ _ _ J 

TRT'=TRT/r5.cr " ■ 

TPLN=TPLN/15.0 

rc=fc7i5.o 

WRITE (6,3) ID, TRT,TPLN,TC 
PUNCH 13, ibVTKT,TPLN7TC 

3 FORMAT! IX, I6,2X,3(F5. 1,1X) ) 

1 3 FORMAT! I6,2X,3!F5;Tn)<l )' 

GO TO 99 
98 CALL EXIT 

END 



o 

0 -. 



o 

Or 

c 



Or. 



O 



o 



o 

ERIC 



>■.1 VJ 
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• • ' « FORTRAN 

■ PROBABILITY TEST KEY PREPARATION 

I C IF-IRS T-CARD-CON-TA INS-CONTROL- PARAMETERS ANO -TITLE. 

C COL 1 1 s STANDARD FORMAT 

C COL 2-3 = NUMBER OF ITEMS COL A = NUMBER OF 

3 X G0L--5-6-=^NUMBER-0F-CARDS—C0L--7-8-= NUMBER OF' 

C COL 9-49 = TITLE 

C DATA CARDS CONTAIN ID IN COL 1-5 A CARD SEQUENCE NUMBER 

4 C A ND-RE-SPONS &S-BAf-&R-I-T&A I ON -GROU P-- M EM BE RG I - C OL -7r 78 - — 

C LAST CARO IN JOB IS BLANK 

C ' LAST CARO IN RUN IS A SECOND BLANK CARO 



RESPONSES 

ITEMS PER CARD 



IN COL 6 



--D1HEA1S-1-O N -A N S ( 7 2■^6■I-f-IT>— < l.a) »-ICS.(-10 ) , ANSPR ( 72 , 6 ) ^ -SANS P 4-7 2-t 6-1, - 
1 AVANS(72,6), TITLdOIf SM<72) 



99 



EN=0 

■DO-. 7 1. 



-Ul-»- 7 2- 



DO 71 J=l*6 
AVANS <ItJ)»0 

ANS-(.l-f4-l*0 

ANSPR(I»J)«0 

71 SANSP <IfJ) =0 ' 

JlEAD-(-5-,.U-JtF-^.-Ja-,--KN.f-KIC^.-ia£-»-.( -TI.TL ( .1).^. -1=1 ,.L0 ) 

KG=KF+1 

GO TO (202tl01)» KG 

JTLL-JRaJ 




( < ANSd t J) t J = liKN) tI=JPt JL) 



H. 






16 . 



366 



203 



17 - 



13 - 



K«1 

IF <KK-1) 32t32»999 

99-9-00-1-24-Xe-I-, KK 

JK=K»KIC 

IF (JL-KI) 42f42,32 

32- J L~K 1 

42 CONTINUE 

READ<5tlll )ID(K),KS<K)t 

-I21-d P«- J L— 4^-1 

CO TO 37 
37 G=ID< I ) 

IJ=-_^G.)._5a»50-r41 

41 CONTINUE 
36 CONTINUE 

-PROGRAM-NOW- WE IGHTS-AND-GOMPUTE S PROBABI L I TI ES 
DO 367 1*1 fKI 
SM(I)=0' 

-410-366- Js-I , K N.~.; — ‘ 

SM(1) = SM(I) + ANS(ItJ) 

IF (SMnn 202f202f203 

-CONTI NUE 

DO 367 J=ltKN 

ANSPR(ItJ) » ANSdtJ) / SMd) 

SANSP-( I-,J ) -=-SANSP-(l »J)-+-ANSPR(I , J) -- 

EN=EN+1 
GO TO 101 

-50- 00-51 I=1-,KI 

DO 51 J=lfKN 

51 AVANS(I,J) = SANSPdfJ)/EN 

DG--4^-I»l-»-KI 

DO 49 J=1,KN 

49 WRITE {6fin) I,J, AVANSdfJ) 



367 



21- 



-PUNGH--115 fKF , -K-^f-KT^r-KK^--KIC f-- ( ’ITL ( I ) ,1*1,10) 

DO 60 1 = 1 ,KI 
DO 60 J=l, KN 

._-60—- PUNCH --k 16r-^-,J , A VANS-t-I^,^-)- — - - 

GO TO 99 4" 

202 CALL EX'T 

1 FORMAT f n . T?. n . 



6o 



o 



17 , 17 , 10a41 



115 FORMAT (II, 12, II, 12, 12, 2X , 12HCONTROL FOR , 10A4) 

111 FORMAT (15, II, 72F1.0) 

112 FORMAT ( 1 5, 1 1 , 72F1 . OJ 

516 FORM'AT (IX, 72, tH-, 72, tX , F5.2) 

117 FORMAT (IX, 12, 2X, 1H-, 2X, 12, 2X , 1H-, F5.2) 



# 









I 



‘ 8 ^ 






10 ' 

U- 

II 

13 ^ 

If 



1$ 



U 



17 



19 



19 
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APPENDIX B 

SAMPLE ITEMS FROM TESTS 
EUCLIDEAN CONFIDENCE 



0 




EMPLOYING 

SCORING 



The student is given a description of an experiment to read (Bandura, 

1968 ). Then he is given the following: 

a. Select the correct answer with certainty on the items which have 
unique correct responses. 

b. Express your honest degrees of confidence on these items which 
do not have unique correct answers. 

c. Do not choose your numbers randomly. If you have no preferences 
for any of the responses, give each response the same numerical 
value, i .e . , 1,1,1. 

If you like all of the answers equally well, mark the answer sheet as 
fol lows : 

a. 3 

b. 3 

c. 3 

If you like none of the answers, mark the answer sheet as follows: 

a . 0 

b. 0 

c. 0 

If you like just the second answer and none of the others ; mark: 

a. 0 

b. 9 

c. 0 

If you like some answers better than others, respond with a digit from 
0“9 which will express your odds on that answer. For example, if you 
like answer c three times as much as answer b, mark: 

a . 0 

b. 3 

c. 6 

A. Answer the following questions according to the instructions. The 
questions refer to the report of the experiment by Bandura and McDonald. 

1. The purpose of the experiment by Bandura and McDonald was to: 

a. illustrate the fact that strong dominant repertoires limit 
the opportunities for reinforcing subordinate responses. 

b. to demonstrate the superiority of modeling over operant 
cond i tion ing. 

c. to test the validity of the stage theory of moral development. 







2. It is important to research the relative efficiency of modeling 
over operant conditioning because: 

a. operant conditioning can be automated. 

b. many desired complex behaviors seldom occur as random 
behaviors . 

c. the excessive emphasis on modeling in Spartan education 
resulted in an increase in homosexuality. 

3. It is important to research the stage theory of moral development 
because : 

a. the morals of today's young people need improvement. 

b. an understanding of the different reactions of subjective 
and objective children may suggest a basis for making 
instructional decisions. 

c. the stage theory has an elegant simplicity. 

4. The factor which differentiates between subjective and 
objective children is: 

a. deviancy, 

b. invariance. 

c. intention. 

5. How well described are the following? State your relative 
preference. 

a. the scheme for classifying subjects. 

b. the treatment. 

c. the dependent variable. 

6. What additional information would you like? 

a. the sex of the subjects. 

b. the age of the subjects. 

c. the IQ.S of the subjects. 

7. What do you like about this experiment? 

a. the random assignment of subjects to treatments. 

b. its pursuit of a new or novel line of thought. 

c. the statistical analysis. 

8. If you were to design a similar experiment you would: 

a. modify the design. 

b. modify the statistics. 

c. modify the treatments. 

9. Relative threats to validity in this experiment are: 

a. history. 

b. regression effect. 

c. instrument decay. 

10. In applying these findings to the classroom, one must be concerned 
with: 

a. the economic cost of modeling. 

b. the external validity of the findings. 

c. the difficulties inherent in applying the results of animal 
experiments to human behavior. 




ItdfflB From 

TiT>T OF ACTJOJIG 

* 

This tecit is doali',nf.'.f! io Foam ii’liai'*ySu r.liould do in certain cituations 

ani how much cortainty you hnvo vihen you clioosc these aetJonn, The jiifori:.a.lion you 
give will be coxrTidontial, 3t will not bo uoed to dotcrniinc any cuade for a coui'sc. 

There are 7 pages. Please check that you have all 7 pages plus an Annvrcr Sheet, . 

This is a timed test. Please wait at tlw5 bottorri of those pages whoi'c it asks you 
to wait, until the explainer says to go on. 

The tost consists of 3.4 incidents. Sets of statements or ansv7(>rs follow those 
Incidents, There may or nay not be a single correct r.ns\:er. The prcci.se response 
which you night desire nay not be listed, Infori.;atlon. th;-.t you night like to have 
may be missing, ilovcvev, for each set of staternents, yo\j are to nark: 

1) the strong ili of vovr vrc^cvf’iico for each statement 'Ti., b, and c". This is 
dona by placing a numbei' from sero to nine (0 to 9) on the lints following 
*'&! b, and c" on the Answer Shoot, A zero indiccates your rejection of the 
statement, A nine i)KUcatc.s that you prefer that statement over the others, 

2) hovf c er tain y oxt, arc of the sot of lesponrico you have made. This is done by 
placirig a number from zero to nine (0 to 9) cm the line follovjing *'d" on 
the Answer Sheet, Zero c o mrlctr.0.y u nsure. Mine absolutely certain - 



For e5ca,raple, if you like only re.sponse "a", you would ir.ruk; 




If you arc partial] y certain of -this set of resporisos, you would markt d, 

If you like "a” not at all, "b" most, and "c” a lesser ai.-iount, you could mark; 

b. 7 • 



c, 1 c , 3 . . c , 4' 

d, 1 d. Z d, 7 

In these 3.ast examples, certainty, marked 1)\ "d", varied fro:*i very little to 
quite a bit. 



Here is an examplfi for you to work; 



Tho nurse who vranis to sugge.st good sources of Vitoaln C 
will inci.ude; 

a, citrus fn!its, 

b, broccoli. 

0, apples. 



ANSKPR SHSFT 

a, 

b. 

0 , 

d. 



The more certain you r.'-, iho higher* your score wil.i. bo if yoi’r rcspon~.es arc close 
to the moan of an*. r, i*. . by expert nurse:}, Houevex*, if your re.spoji:;o.s "a, b, 
and c" are far fvc:-i of ans}.*ors given by Ujc experts, a low certainty, as 

marked in "d" v;ili vt ' lower* ncoi'c. 

The numbor.s to t)je . .hie lines on the /.n.:;ucx* Sheet arc for da:Ux procer.sing 

on3.y* 
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INUlDtiNT Ai Ml', Frc-nk Muliry, cv 7?' yciir old retired r,:ichinlst, viaa helpitig hi.« 
wife carl weed;..; to ihu go.x'Ik’.^o can when he onddnily rj.ipp:,;d and fell. He was 
brought to tl\e hoopitol by aiiibulence, ar4 over the p.x&t day, ho has been comatose, 
Kls reparations arc stertorouG, a.nd the right cldo of his mouth blows outward with 
each ej^ration. His right leg has a tendency to fall into ertcrno.l rotation, A 
lumbar puncture sho\ro bloody ccrclvospiival fluid, Hir.; \»;\fe says he has been 
treated by Dr, Adaj:s fo3' hypertension for 20 yeo-rs. His tentative diapnosio: CVA, 

His admission chest >:-ray shoi.’s a markod hypertrophy of the left ventricle i 
Dr, Adams' orders include; Mcrcuhydrin 1 ml, IN q,o,d, 

500 ral, 5</1 D/v.' 

1000 ml, 3,actated Hingero IV 

1, Essential in the care of Mr, Buttry at this tine is (arc); 

a, regulating IV fluids for x'apid re plcnlshr.ont of electrolytes and of glucose 
to the brain, 

b, leaving a pa.dded tongue blado at the bodsidc, 
c« suctioning; oropharyngeal secretions, 

2, During Mr, Buttry 's first few days in t)iu hospital, a small elevation of his 
temperature, pulse, and respirations, nay signal 

a, infection or dehydration, 

b, progrcaslvc deterioration of the centers of the brain, 

c, a normal course. 



3» Preventing deformity is an important part of the nurse's responsibility to Mr, 
Buttry. A full range- of-mot. ion cxereisc; progrrsa should bo etax'tcd as eoon as 

a, Mr, Buttry awakeijs and starts viovlng in bod, 

b, Mr, Buttry is ordered to be "up in chair," 

c, Mr.. Buttry's blood prosswre, pulr.e, ai'-rl I'cspirations sta.blIii>o, 

IMCIDEMT B; Mr, Buttry oponed hirs eyes durjuj the third day of hla hospitallf'-ation, 
Althovigli unoblc to spof.h intoli.iglbly, he grasped his wife's £u'm ;r.ith recognition. 

He responded to instructions to wove, albeit slu;vg5.t;hly. He is able to ovial3,o\r dps 
of water. 



4, At this point j.n hi.s recovex-y, the nujVie.s shouD.d assume that self-help training 
should include 

a, turni.irg from side to side, 

b, sitti)ig up, 

c, feeding liii-.self , 



5» The mvrses should also 

a, phono D.c, Adans, 

b, encourage Mr, iiuttry to talk, 

c, rcfi'a’in from asking Mr, Bui try que ."-Mons,, 

INCIDENT C; On the dth day, a liour after Mr, Buttry 'o bed hr.d been rolled dowji 

for the night. Hr, J3uitry nlrrtt.d ccngh3.u>:;, Althoogli hin cough was infrequent. It 
was bothersone enough to l.ecp liiu c.W'd:c, The nigl-i nus.se found him still awake at 
11 j 30 peM, Ko ho.d received Kcrbuial gr iii at K> p,).''. In addit.ion to tiii%, order for 
Nembutal at hoS,, th:*re is a fv/rthcr ordt r written; "May c;i.vo Noi:‘butal gr ins once 
during the night fox* nloop p,r,n," 

6, The night Tun::>c siiCuld 

a, phone Dr, /ul.rafj , 

b, give a warm ch'ink of ni.lk r.).;.'!. rnct.lica' pillo’.?, and assure him that he will to 
given another el^ vpijv; p.ill ri; -sooxi am it is all right to do so, 

c, b.lm ?■ c.f f.v non- -ir. reset ic dough syrv.pc 



O 
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INCIDENT Di Mr« Buttry’s ovcnlns routine Cind i,-odlcation were soon under control, 
A, few' days D.ator, hoiicvo*!', Mr, B*e reopJratlono sounded gurglini.:. He looked pale 
and was orthopncic. Blood chemistry test results (cb.-awn the evening "beforo) 



A chect x-ray indicated Increosed vasculax'lty of the lung fields. Dr, Adams 
ojrdercd Morplilno SC^ 15 *H* given stat. He thou gave Oualain 0,5 mg IV and 



Aldactone 25 Dtg, q,i,d, 

Digitoxln 0,6 mg, stat and in 6 homrs 
then Dlgltoxln 0,2 mg, q. 6 hrs, 

1 Cm, Sodium diet. 

Rotating tourniquets on 45 min,, off 15 min (four extremities) 
Oxygen per cannula at 2 to 7 liters/nin, p,r,n. 



?• . The xnxrse should asou.rae that Dr, Adams must be notified by phone If 

a, Mr, But try becomes nausoated or has abdominal crainp-s, 

b, Mr, Buttry's extremities become uncomfortable and discolored, 

c, Mr, Buttry*s face becomes pJ.nk and flushed, 

8, Tho nurse should te.ll Mr, But try that 

a, coffee and tea should bo avojdcd, 

b, meat, fish, and poulta:y will be limited on his diet tray, 

c, if ho does not Irilce come food on his ti’y, she will get a substitute, 

9* Tho nurse will further Mr, Buttry’o recovex-y by 

a, using a massager on his posterior chest \;all, 

b, sttggcsting he turn touard the loft side when eating, 

c, giving oxygen at 7 litox's on evidence of dyspnea, 

10, The nurse notes thiit Mr, B*s m-lne output suddenly increases substantially 
with this regime. She should 

a, expect that labox'atory reports vfill shovr a furthex' decrease iti potassiwn, 

b, expect a greater likelihood of anorexia and gastx'ciniestinal complaints, 

c, recogniise this as a guarded prognostic sign. 




ordered! 



Stop Mercuhydrin 



STOl', I’LIiASI'^ DO KC/r TUHN THIS PAGE UNTIL TOLD TO DO SO, 
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Saapl* It«u Trom 

T«st of TrottblosoM Oroup KaaMBont 81t«atloas 
Forn II SalUo S* QrarehiU 



This study is concerned with actions taken by social isiroup workers 
in group meetings and with their reasons for these actions* Comparisons 
will be made among several schools of social work and between beginning 
and second year students*. 

There are two parts to this questionnaire: 

PAkT I ; You will be given a description of a boys' group and a 
description of a men's group* Following eacli group description 
you will be given a set of three incidents which occur during 
meetings of that group* 

You will be ashed to rate several actions that a group 
worker might select as possible responses to the incidents* You 
will also be asked to rate several reasons that a group worker 
might offer for his actions* In each instance, you will be asked 
to indicate how certain you are about your ratings* 

In addition you will be asked to rate each set of worker 
actions and worker reasons in the way you think your first year 
field instructor would rate his preference and (2) the way you 
thin): your first year social group worl: method teacher would rate 
his preference* You will also be asked to indicate how certain 
you are about your estimates* 

The answers to PART I will bo recorded on a separate 
answer sheet* 

PART II ; You will be asiced to give some information about yourself 
your education and your work experience* 

As a researcher I am interested in learning how groups of students 
respond to the items in the questionnaire, rather than how any specific 
individual responds* Therefore, I will not request tliat you put your name 
on the questionnaire* However, the Information will bo considered confiden 
tial and the responses of each of you will be available to tlie researcher 
only* 

It is expected that it should take between one and two hours to 
complete this questionnaire* 



Please turn to the Instructions on the Use of the Answer Sheet 



Form II 



INSTRUCTIONS FOK USING HIE ANSWER SHKET 
1. Record your own preferences under Self Rating. 

2* Record how you think your first year field instructor would rcsi>ond under 
Field Instructor . 

a. If you had more than one field instructor in your first year, 
select either; 

The field instructor whom you had for the longest time; 
or if you had field instructors for equal lengths of time, 
select the field instructor whom you had during your last 
quarter or semester. 

3 * Record how you tiiink your first year social group work method classroom 
teacher would respond under M et hod Teacher . 

a. If you had more than one group work method teacher during your 
first year of social group work education select cither; 

The teacher whom you had the longest time; 

or 

The teacher who carried the major responsibility for 
your first year method class; 

or 

The tuiichcr whom you had during the last semester or quarter 
in your first year. 

4 . Place a digit from 0 to !) in the blank opposite A, 11, or C. Each number repre- 
sents the strength of your preference for each response. Ilov.ever, the sum of 
the throe responses should add up to 9. Zero indicates rejection, 9 represents 
maximum preference. 

If you liked only response A, you would answer A 9 

B 0 
C 0 

If you liked all three answers equally, you would mark - A3 

B 3 
C 3 

If you liked answer B most, C a small amount and A not at all, you might 
mark - A 0 0 

D 7 or 8 

C ^ 

The number you write down should be proportionui to your preference. 

THERE ARE NO CORIffiCT ANSU'ERS 

After marking your preference for A, B, and C, mark in square 1), a 
number from 0 to 9 which tells liow certain you are of your distribution 
of preferences for A, B, and C. 

0 s Completely unsure 
9 = Absolutely certain 



Turn to the FIRST GROUP DESCRIPTION. 



Cy V 
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BOYS* GROUP 



Please read the following description of Tim BOYS' GROUP and the GROUP MI^IMBRRS. Spend 
aa much time studying this material as you wish. When you urc ready « turn to the group 
incidents. If you wish you may refer to the group and/or individual descriptions as 
you rate worker actions and/or worker reasons. 

Setting : Children's Unit of a State Mental Hospital 

Timing; The group has been meeting with a social group worker once a week 
for two months. 

Group Description ; 

All group members, aged 7 to 9, were admitted to the hospital because of serious 
behavioral problems which prevented attendance at school, and/or retention in 
an open community. 

When the group was formed all the boys had made considerable progress in the 
closed residential setting and the hospital was considering discH,arge. A 
group worker was assigned to these boys to assist in their resocialization to 
community living. The group service was planned to assist the boys to increase 
their attention span, to increase their impulse control and to learn appropriate 
social skills (including interactional and physical components of games). The 
boys wont the group to serve as a means of having fun, getting off the ward, 
and getting away from the "crazy" kids. 

The Worker thinks the group members are benefitting from the group’ experience 
and usually the group meetings proceed in a constructive manner towards stated 
goals. In most ways the group can be considered a good group, with members and 
worker successfully working together. 

Individuals 

Aaron , Large for his age, poor coordination, easily loses his temper and 

destroys equijwnent. Steals other boys' possessions. 

Barry. 9; Very passive; cries easily. Often angers other members by refusing 
to do something, such as take his turn in a game. May become irrationally 
angry and lose all control. 

Danny, Th '. Subject to epileptic seizures. He threatens to have seizures to 
control group. Grimaces and acts "crazy" to get attention. Hyperactive. 

Eddie, 8; Leader of the group; good athlete and very bright. He has little 
frustration tolerance and will bite anyone close to him when he is upbet. 

Often tries to challenge adult authority. Upset by worker's attention to 
other group members. 

Gary. Very immature; sucks thumb, frequently masturbates. He clings to 

worker while being verbally abusive to other members. Scapegoated frequently. 



When ready, turn to INCIDKNT ONE.. 



O 
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Porro II 



BOYS* * GllOUP: INCIDENT ONE 



Please read the following incident: 



All of the members were playing a fairly rough, but controlled 
game of "King of the Mountain." Barry and Gary seemed to have teamed 
up to hold off the "attack." Eddie soeiiied quite surprised when the 
weaker members held up against him. Suddenly Eddie bit Gary. Gary 
gave up, fell off the mountain, began screaming, "I quit!" Eddie im- 
moadiutoly began taunting, "Baby, baby, where's your bottle? Where is 
your diapers?" Tlic sing-song chant took hold mid all five boys shouted 
it loudly at Gary, apparently ignoring tliut he had been bitten. 



RATE THE riRST OE E T. 'EVTC T\' TO THE ABOVE INCIDENT IN THE BOYS' 

GROUP. 

I. ON THE ANSWER SHEET, IN THE SELE RATING COLUMN, INDICATE IHE STRENGTH OF YOUR 
PREFERENCE FOR EACH WORKER ACTION . 

II. ON THE ANSWER SHEET, IN O'ilE FIELD INSTRUCTOR COLUMN, RATE HOW YOU THINK YOUR 
FIRST YEAR FIELD INSTldJCTOR !('OULJ) RATE THE STRiiNGTII 01*’ HIS PREFERENCE FOR EACH 
WORKER ACTION. 



III. ON THE ANSWER SIHOET, IN THE MliTHOD TEACHER COLUMN, RATE ItOV/ YOU THINK YOUR FIRST 
YEAR GROUP !/0 iv‘ K METHOD TllACHlilR WOULD KATE THE SXIiliNGTH OF IIIS PREFERENCE FOR EACH 

woidvicR action'.' 



IV. IN EACH INCIDENT INDICATE ON LINE D HOW CERTAIN YOU ARE. 



* A. The group v»'orkcr hclj>s the group quiet down and reviews the nature of the 

group interactions th.it have ,iu.st occurred. He oncoiirnges the group members 
to discuss their feelings toward Gary. 

B. Tlie group worker stops the game and gathers the boys together, lie tells them 
that he will not permit the group to tease Cary for crying when he has been 
bitten, nor will ho permit the boys to bite each other. 

C. The group worker comforts Gai*y. 
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A. The group worker sends Eddie out of the group for biting Gary. He discusses 
with the group that boys usually like to play rough but they must play fair 
and squ.aro. 

B. The group worker reminds the group of its contract, pointing out that Gary's 
crying, as well as Eddie's biting when he is upset, are behaviors which the 
group might help the boys chungc. 

C. The group worker reminds the group of its contract and encourages the group 
members to discuss what has taken place and how this may affect the function- 
ing of the group. 



If you wish to make any comments about the Worker Actions in response to this incident, 
write them below. (OPTIONAL) 



Form II 
Boys’ Group 
Incident One 



I. ON THE ANSWER SHEET, IN THE SE LF InWTING COLUMN, INDICATE THE STRENGTH OF YOUR 
PREFERENCE FOR EACH REASO N WHICH A GROUP WORKER MIGHT OFFER FOR HIS ACTION. 

II. ON THE ANSWER SHEET, IN THE FIELD INSTRUCTOR COLUMN, RATE HOW YOU THINK YOUR 
FIRST YEAR FIELD INSTRUCTOR WOULD RATE THE STRENGTH OF HIS PREFERENCE FOR EACH 
REASON. 

III. ON THE ANSWER SHEliT, IN THE METHOD TE/\CHER COLUMN, ILM'E HOW YOU THINK YOUR FIRST 
YEAR GIWUP WORK ME’DICO TEACHER WOULD RATE THE STIUWGTH OF HIS PREFERENCE FOR 
EACH REASON. 



IV. IN EACH INCIDENT, INDICATE ON LINE D HOW CERTAIN YOU ARE. 



JJ 

A. The extent of aggression and/or liostility expressed in a group should be con- 
trolled by the group worker, 

B. A group member who is excessively teased or hurt needs comforting and/or 
support. 

C. A group may need help in identifying and dealing with obstacles which 
prevent the group from jtrogressing towards the group goal. 



4 

* A. A group member who intentionally hprts another member physically or psycho- 
logically, should be disciplined, regardless of whether the hurt member had 
provoked the aggression or not. 

B. The group worker should not attempt to retain control of the interactions be- 
tween or among the members, except in incidents of possible physical danger 
to a member. 

C. When it appears that group members have lost control of themselves, so that 
their behavior is neither liclpful to themselves nor to the other group members, 
the worker should act so as to restore control. 



If you wish to make any comments about the Reasons for Worker Actions in x*esponse 
to this incident, write them below. (OPTIONAL) 



Go on to the next INCIDENT 



Form II 



BOYS’ GIJOUP; INCIDENT TWO 



Please read the followinf? incident; 

All the momiiers hud been workiri.'i- on pluslic airplane models. iThe 
group was sliaring throe tubes of glue- As the meeting time was up and 
the grou]) began to clean up, the worker noticed one tube of glue was 
missing. This has ha[)pcned frequently^ 

The worker asked the group members if they could locate the tube 
of glue. Immediately Datmy said Aaron had swiped it. Aaron uneasily 
put tlie tube of glue back on the tabic. Danny yelled, "I was right, 
Aaron liad it. He's a little thief?" Gary said, "Our grouj) doesn’t 
need any dirty thieves. Does anyone want him in the gronp?" Sudden- 
ly Eddie demanduU silence- "It's our gioiip; lot's vote on it." All 
the boys shouted at once, "’li'o don’t want liim." Eddie declared, "It's 
five against one . , « he's out." 



RATE THE NEXT RlUIl SETS OF STATEMENT'S (H-S) 
BOYS' GROUP. 



IN EEMiUI'.NCE W THE ABOVE INCIDENT IN THE 



I. ON THE ANSWER EiHEET, IN THE SELF HATING COMJMN, 
PREFERENCE HIR EACH UOIJUIHI ACTION, 



INDICATE THE STRENGTH OF YOUR 



II. ON THE ANSWER SIIEETi IN THE- FIELD INSTRUCTOR COLUMN, RATE HOW YOU THINK YOUR FIRST 
YEAR FIELD INSTRUCTOR WOULD RATi: THE STRENGTH OF IllS l'l?EFKIH:.NCE FOR EACH V.0RUFJ1 
ACTION . 

III. ON THE ANSWElj SHEET, IN THE METHOD TEACHER COLUMN., RATE HOW YOU THINK YOUR FI i;ST 
YEAR GROUP WORK METHOD TI;aCIIE 1{ WOULD 15ATE THE STlH'bNGTH OF HIS PREFERENCE 
WORKER ACTION, 



IV. IN EACH INCIDENT INDICATE ON LINE V HOW CERTAIN YOU ARE. 



A. The group worker asks the boy.s, to think about their quick vote. He adds that the 
boys seemed awfully angry and did not seem to bo thinking about wliat they were doin 
He asks if the group really wants to lose one of it.s members. He accepts the final 
group decision. 

B. The group worker says that the decision iibout who is a member of the group is the 
worker's and tliat nobody could be voted out of the group by gi-oup action. 

C. The worker asks Eddie if he really thinks that Aaron is a thief. He tells him 
that Aaron had returned the glue and that Aaron is not a thief. 



6 * 

* A. The group worker helps the group members tell Aaron how they foci about his be- 
havior, in this meeting and others. He helps Aaron respond to what tJie boys say. 

D. The group worker comments that all the boys have concerns about problems like 

stealing. The boys can talk about these concerns in the meeting but they may not 
kick anyone out of tiie group. 
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C. The group worker tolls Eddie that he is upsetting everyone by yelling for the 
group to vote a member out; he tells the boys to calm down and finish cleaning 
up. 
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Form II 
Doys * Group 
Incident Two 



2 



If you wisli to make any comments about the Woz'kcr Actions in response to this incident, 
write them below,. (OPTIONAL) 



A. The fjrouj) is a mutual aid system which must remain free to make its own decisions. 

B>. The group is used .as both moan.s and context of the tree tment of the individual; thus 
membcr.ship is determined by the worker in relation to individual needs. 

C. The decision by tfio p,rou[i mcmbcMV? to exclude a member from the group must be based 
on rea.sons which .seem valid to the work<M'. 



Ao The responsibility for each member's welfare belongs to the group worker. Tlierefore, 
the group worker retains a veto power over all groiij) decision.^. 

B. The group worker sliould correct tiic members when they express wrong ideas or feelings 
in order to maintain an accurate awarc*ne.s.s of reality within the group situation. 

C. The use of authority by the worker can become a central stumbling block in the 
helping relationship. 



If you wish to make any comments about the noasons for k'orker Actions in response to this 
incident, write them below. 




7 . 



8 . 



) . 



3o on the the next INCIDKNT 
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CRITERION RESPONSE DISTRIBUTION PLOTTING PROGRAM 
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Criterion Response Distribution Plotting Program 



This program plots the distribution of responses assigned to 
each item on a test. For convenience, the probability assigned to 
each option is rounded to the nearest tenth. The response triangle 
can then be considered a set of 55 discrete points, with each point 
corresponding to a specific set of probabilities. The program 
calculates the probabilities assigned by each subject to each item 
(rounded to the nearest tenth) and performs a frequency count for 
each response set. Output is in the form of a plot of the response 
tr i angle. 

Program limitations 

The program will plot responses for up to seventy-two items. 
All items must, have three options. 

Card setup 

C^rd 1 Title card (cols 2-60) 

C a rd 2 
Cols . 

1 Punch a 1 in this column 

2-3 Number of items 

5-6 Number of cards for each subject (Not more than ten) 

7-8 Number of responses per card 

A sample of the output and the program listing follow. 









XEQ 


« 


• 


Vi 


FORTRAN 

CRITERION RESPONSE DISTRIBUTIONS 




X 




DIME'JSION AJ03(12), ID(l'), KSIlOIt 


ANS(72,3). ANSPRI72.3I, SN( 


_ _ 




1), JSTOREI/? ,1 1 , 11 ) 
JOB = 0 




c 


937 


JOB = JOB + 1 

WRITE OUTPUT TAPE by 986 1 JOB 




l_ 


■ 


READ Input tape 5,2235 , (ajo8(D,i=i 

WRITE OUTPUT TAPE 6 , 22 35, ( A JOB ( I ) , I = 


,12) 

1.12) 


7 




00 4000 I = 1,72 
no 4"'01 J = 1,3 






4'^: 1 


ANSI I, J) = : 
ANSPR ( I, J ) = ^ 
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00 4C0P JA = 1,11 
00 4:0'^ J3 = 1,11 






4 O ■J 

241 


J STORE (I , JA,JB) =0 

READ INPUT TAPE 5,1, KF 


,K I ,KN,KK, KI C 


n . _ 




WRITE OUTPUT TAPE 6,93C,KI 
IF (KF )94?,940,211 




ir 


211 
1’' 1 


GO TO (1'1,102),KF 
JP=1 




13 

t< _ 


999 


K = l 

IF (KK- 1) 32, 32,999 

DO 121 K=1,k"k . 

JL=K^ KIC 





1? 


32 


IF ( JL-KI )42,42,32 
JL=KI 




_ ... 


^2 


CONTINUE 

READ INPUT TAPE 5 tl 1 1_, I D £ K}.j, K S (K ) , 


1 (_AI^ S.( J. J J )._fc J =1. J .K N ) , I r J.P , J L ) 


»*• 


121 


IF ( ILMK ) )940 ,240 , 121 
JP=JL+1 




la _* 


1^2 


GO TO 37 

JP=l 








K.= 1 

IF (KK-1 ) 532,532,998 






993 


00 172 K=1,KK 

.__..JL = K.:vKyJ.Q...__ 




1 

SI . __ 


532 


IF (JL-KI ) 533,533,532 
JL = KI 


' 




533 


CONTINUE 





':j 



IF( I0( K) )9^+C,240,172 

17.2.._J_P=J_L+_1 

3 1 G = I D( 1 ) 

EN = E N»I;- 

N=EN 



ij _ J F J c; ) 9 £»rii 2 '+.O.,. 4 1 _ 

41 CnNilNUE 

,3 I. f.( KK.r.L ) .....3.6 I 3.6.i8A__ 

81 CONTINUE 

3 dii_36 k^2,kk ;; 

LA=in('K) 

Lrt= ID! ... 

IF(LA-LB)33",30 »33 

S.C. „L C =.K S (_K). 

► LD = KS(K--l ) 







IF( LC-LD) 35,35,36 


... .. 




33 


WRI TE OUTPUT T APE 6 ,.1 1 3 , ID I K ) 




■ 




CALL EXIT 




O 


35 


WRITE OUTPUT TAPE 6,114,ID(K) 


L 

po 

1 

1 

1 

1 

I 

1 

1 

1 

1 

I 

1 

I 

I 
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CALL EXIT 


36 

C 


CONTINUE 

PROGRAM NOW WEIGHTS AND COMPUTES PROBABILITIES 


D.3 367 I = l,KI 
SM(n=3 


366 


00 366 J=1,KN 
SMI I ) = SMI I ) + ANSI I , J) 






B7RK= SMIIJ 

IFIRZRK) 5327, 5327, 5320 




532 7 


ANSPKt I , J> = O. 
GO TO 367 




5328 


CONTINUE 
00 367 J=1,KN 




367 


AN'SPRII,J)= ANSI I, J) / SMI I ) 
CONTINUE 




no 5'' 13 I = 1 ,KI 
JA = ANSPRd fl) ^ lO. + 1. 


5 '' ] 3 


J3 = ANSPRI 1 ,2) K. + 1. 

JSTOREIl ,JA,JB) = JSrOREI I , JA, JB) 1 




240 


GO TO 211 
CONTINUE 




5'' 14 


I = C 

1 = 1 + 1 




5C15 


IF II-KI) 5015,5015,987 
WRITE OUTPUT TAPE 6, 5C16, I 




50 1 6 


FORMAT I41H1CRI TERION RESPONSE DISTRIBUTION 
WRITE OUTPUT TAPE 6, 500 1, JSTORE I I , 1 1 , 1 ) 


FOR ITEM, 15) 




WRI TE OUTP'uf fAPE 6, 50C2, I JSf ORE 1 1 , 10 ,"J ) 
WRITE OUTPUT TAPE 6, 500 3, IJSTOR El I ,0 , J ) , 


, J=l,2) 

„j=x;»3L 




WRITE OUTPUT TAPE 6, 5004, I jsfORE I I , 8 , J ) , 
WRITE OUTPUT TAPE 6, 5005, IJSTORE I 1 , 7 , J ) , 


J= 1,4) 
J=1.5) 




WRITE OUTPUT TAPE 6, 5v006, IJ STORE 1 1 , 6 , J » , 
WRITE OUTPUT TAPE 6 , 5007, IJSTORE 1 1 , 5 , J ) , 


J=i ,6 ) 
J=l,7) 




WRITE dutPUT TAPE 6 , 5008, UstORE I I *4 , j ) , 
WRITE OUTPUT TAPE 6 , 5009, I JSTURE I I , 3 , J ) , 


J ® 1 , 8 ) 

. j=i,9) : 




WRITE OUTPUT TAPE 6 , 5010, I J sfoR E I I ,2 , J ) , 

WRITE OUTPUT TAPE 6 j_ 5011, I J STORE 1 1 , 1 , J.).» 


j=ii, 10 

..JrAiLU - 


• 


WRITE OUTPUT fAPE 6 , 5012 

GO TO 5014 




940 

5cn 


CALL EXIT 

FORMAT! // 6 CX, 4H*10*, 5X, 1 I 5, 5X, 3H*0*) 




5002 

5003 


FORMAT I//59X, 3H+9’?- , 5X , 215, 5X , 3 H>t-lV) 
FORMAT I//57X,3H*8''=, 5X, 315, 5X, 3HX= 2* ) 




5004 
50.' 5 


FORMAT I//55X', “3H'^'7+“, 5X, 415, 5X, 3HX=3*) 
FORMAT I//53X, 3H*6'- , 5X, 515, 5X, 3H«^4*) 




5006 
500 7 


FORMAT I// 41X, IHA, 9X, 3 Ht 5 X , 5X, 615, 5X 
FaR^'AT I //49X, 3H’i'4''' , 5X , 7 15, 5X, 3H^6*) 


5* , 9X , ”’i HB ) 


5008 

50C9 


FORMAT I//47X, 3H«3+, 5X, 815, 5X , 3H<= ) 

FORMAT I//45X, 3 H+ 2 -, 5X, 915, 5X, 3H*8>p) 


■ # 


5010 

5011 
5tf2' 


■ FdRMAf'T77"43XT'“3H’^^ '5X, 1015 7 5X , ) " 

FORMAT I//41X, , -5X ,. IJ J 5> 5X, 4H*10*)_ 

FWMAt'T//50X7 54H * 9 ^ aq,;? " ^^' 7 * * 5 * 

11 + i-0’^///60X, IHC) 


■ "xT 5 “* • ' * 3*= * 2 


93C 

, 1 


FORMAT I 2 X , I 3 , 1 X, 6 H I T EM S , IX , I 1 , 1 X , 1 9H RESPONSES PER ITEM! 

FORMAT II 1, I 2, 1 1 , 12, 12 » . 


— Yj--| 

112 

: •113 


FORMAT I 15, 11, 

FORMAT! I 5, 11 ,36F2 .0 ), 

FOR MAT! 35H ERROR I N» ’i o' CODES'; FdR"SU37eC7"Nn^^ 


■715)7 


C 
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